Weather image classification method

Abstract

The invention discloses a weather image classification method, belongs to the technical field of weather image classification, and is used for improving the precision of weather image classification. The method comprises the following steps: expanding 3 channels of a first weather image to 32 channels; performing convolution processing on the expanded first weather image by using a top-to-bottom and bottom-to-top sequential scanning mode of double channels to obtain a first feature map and a second feature map output by double-channel convolution processing; fusing the first feature map and the second feature map to obtain a third feature map, and expanding the channel of the third feature map to 1024 channels; processing the expanded third feature map through a channel self-attention mechanism to obtain a fourth feature map; and classifying the fourth feature image by using a classification block of a YOLO framework to obtain a classification result of the first weather image.

Description

Technical Field

[0001] The present application belongs to the technical field of weather image classification, and specifically relates to a weather image classification method. Background Art

[0002] Image classification technology mainly uses different features reflected in image information to quantitatively analyze images and classify them into certain categories. YOLO is an existing mature image classification technology. The YOLO framework was first proposed by Joseph Redmon in 2016 to transform the object detection problem into a regression problem. Early image classification algorithms such as R-CNN and Fast R-CNN usually use a region-based approach to classify images, which is computationally intensive and slow. YOLO, on the other hand, greatly improves the speed of image classification by dividing the entire image into multiple grids and directly predicting the category in each grid, making it suitable for real-time applications such as autonomous driving, surveillance, and robotics.

[0003] YOLO's classic models include: YOLOv1 (2016): The first version proposed, which uses a regression model to predict the category of objects, significantly improving the detection speed, but with lower accuracy. YOLOv2 (2017): Also known as YOLO9000, it introduced batch normalization, multi-scale training, Anchor Box and other technologies to improve accuracy and stability. YOLOv3 (2018): Further improved the network architecture and introduced a multi-scale feature pyramid structure, which can better detect objects of different scales and reduce the problem of missed detection of small objects. YOLOv4 (2020): Proposed by Alexey Bochkovskiy, it introduced the latest technologies such as CSPNet and Mosaic data enhancement, greatly improving the performance of the model while retaining efficient real-time performance. YOLOv5 (2020): Launched by Ultralytics, although the name is not exactly the same as the previous YOLO series, it is developed based on the PyTorch framework, further optimizing the network structure, data enhancement and training strategy, and is widely used in the industrial field. YOLOv6 & YOLOv7 (2022): These versions were proposed by other teams, focusing on high accuracy while maintaining low computational cost, further promoting the application scope of the YOLO algorithm.

[0004] However, the above processing methods have not been applied to weather image detection and processing. Therefore, the use of traditional machine learning algorithms for weather image classification still has the problem of low detection and classification accuracy. Summary of the invention

[0005] The embodiment of the present application provides a weather image classification method, which can solve the problem of low detection and classification accuracy in traditional weather image classification methods using machine learning algorithms.

[0006] In a first aspect, an embodiment of the present application provides a weather image classification method, the method comprising: expanding 3 channels of a first weather image to 32 channels; performing convolution processing on the expanded first weather image using a dual-channel top-down and bottom-up sequential scanning method to obtain a first feature map and a second feature map output by the dual-channel convolution processing; fusing the first feature map and the second feature map to obtain a third feature map, and expanding the channels of the third feature map to 1024 channels; processing the expanded third feature map through a channel self-attention mechanism to obtain a fourth feature map; and classifying the fourth feature map using the classify block of the YOLO framework to obtain a classification result of the first weather image.

[0007] In the second aspect, an embodiment of the present application provides a weather image classification device, which includes: a first expansion module, used to expand the 3 channels of the first weather image to 32 channels; a first processing module, used to perform convolution processing on the expanded first weather image using a dual-channel top-down and bottom-up sequential scanning method to obtain a first feature map and a second feature map output by the dual-channel convolution processing; a second expansion module, used to fuse the first feature map and the second feature map to obtain a third feature map, and expand the channels of the third feature map to 1024 channels; a second processing module, used to process the expanded third feature map through a channel self-attention mechanism to obtain a fourth feature map; a classification module, used to classify the fourth feature map using the classify block of the YOLO framework to obtain a classification result of the first weather image.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In an embodiment of the present application, the 3 channels of the first weather image are expanded to 32 channels; the expanded first weather image is convolved using a dual-channel top-down and bottom-up sequential scanning method to obtain a first feature map and a second feature map output by the dual-channel convolution processing; the first feature map and the second feature map are fused to obtain a third feature map, and the channels of the third feature map are expanded to 1024 channels; the expanded third feature map is processed through a channel self-attention mechanism to obtain a fourth feature map; the fourth feature map is classified using the classify block of the YOLO framework to obtain a classification result of the first weather image, which can improve the classification accuracy of weather images. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flow chart of a weather image classification method provided in an embodiment of the present application; Figure 2 It is a schematic diagram of a flow chart of weather image channel expansion provided by an embodiment of the present application; Figure 3 It is a dual-channel multi-scale YOLOV8 architecture diagram provided in an embodiment of the present application; Figure 4 It is a flowchart of performing convolution processing on the expanded image using a dual-channel top-down and bottom-up sequential scanning method provided by an embodiment of the present application; Figure 5 It is a flow chart of fusing two channels of an image and expanding the fused channels to 1024 channels provided by an embodiment of the present application; Figure 6 It is a schematic diagram of a process of processing an expanded image through a channel self-attention mechanism provided by an embodiment of the present application; Figure 7 It is a flow chart of classifying a weather image using a classify block of a YOLO framework provided in an embodiment of the present application; Figure 8 It is a schematic diagram of a process of classifying weather images through a neural network provided in an embodiment of the present application; Fig. 9 It is a schematic diagram of a process of iteratively training a neural network provided in an embodiment of the present application; Fig.10 is a structural schematic diagram of a weather image classification device provided in an embodiment of the present application; Fig.11 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0013] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0014] The weather image classification method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in combination with the accompanying drawings.

[0015] Figure 1 A weather image classification method provided by an embodiment of the present application is shown, and the method can be executed by an electronic device, and the electronic device may include: a server and / or a terminal device, wherein the terminal device may be, for example, a vehicle-mounted terminal or a mobile phone terminal. In other words, the method can be executed by software or hardware installed in the electronic device, and the method includes the following steps: Step S102: Expand the 3 channels of the first weather image to 32 channels.

[0016] Specifically, for the input first weather image that needs to be checked and classified, the 3 channels of the first weather image need to be expanded to 32 channels.

[0017] In one implementation, the expanding the 3 channels of the first weather image to 32 channels includes: selecting adjacent samples of each category label according to the category label of the first weather image to obtain multiple category data samples; performing 32 convolution operations on the first weather image of each category in the category sample data, wherein each convolution operation uses a different convolution kernel and bias item; and obtaining a 32-channel output result of the 32 convolution operations.

[0018] Specifically, Figure 2 As shown, expanding the 3 channels of the first weather image to 32 channels includes the following steps: Step S11: inputting the category label of the first weather image data, performing sample selection on adjacent samples of each category label, and obtaining a plurality of category data samples; Step S12: Perform 32 convolution operations on the first weather image of each category in the category data sample, and obtain single-channel images of the same scale but different data by selecting different convolution kernel sizes.

[0019] The convolution calculation formula is as follows: ; ; ; Among them, O(i, j, c): the value of the cth channel of the output feature map at position (i, j); I(i+m, j+n, d): the value of the dth channel of the input image at position (i+m, j+n); K(m, n, d, c): the weight of the convolution kernel, m and n represent the spatial position of the convolution kernel, d represents the input channel, and c represents the output channel. b(c): the bias term of the cth output channel. k: the size of the convolution kernel. For example: if the convolution kernel is kxk, then k is the width and height of the convolution kernel. i and j represent the spatial coordinates of the output image. The convolution result is calculated by traversing every pixel position of the image. d represents the channel of the input image (such as the three channels of RGB), where it ranges from 0 to 2 (for a three-channel image).

[0020] Step S13: Calculate the single-channel output of different convolution kernels according to the formula of the Flower Girl Book.

[0021] Repeat step S11 to step S13 32 times, using different convolution kernels K and bias terms b each time to generate 32 different output channels to obtain 32-channel output results; Step S104: performing convolution processing on the expanded first weather image using a dual-channel top-down and bottom-up sequential scanning method to obtain a first feature map and a second feature map output by the dual-channel convolution processing.

[0022] In an embodiment of the present application, it is necessary to perform convolution processing on the expanded first weather image using a dual-channel top-down and bottom-up sequential scanning method to obtain a first feature map and a second feature map output by the dual-channel convolution processing.

[0023] In one implementation, the first weather image after expansion is subjected to convolution processing by using a dual-channel top-down and bottom-up sequential scanning method to obtain a first feature map and a second feature map output by the dual-channel convolution processing, including: performing a top-down convolution processing on the first weather image of the expanded 32 channels to generate the first feature map; performing a bottom-up convolution processing on the first weather image of the expanded 32 channels to generate a second feature map.

[0024] Figure 3 The dual-channel multi-scale YOLOV8 architecture diagram provided in the embodiment of the present application is shown in FIG. Figure 4 The convolution processing of the expanded first weather image using a dual-channel top-down and bottom-up sequential scanning method includes the following steps: Step S21: Set the convolution step size of the convolution layer to 1 and the convolution step size of the pooling convolution layer to 1.

[0025] Step S22: performing top-down convolution processing on the expanded 32-channel first weather image to generate a first feature map.

[0026] Step S23: performing bottom-up convolution processing on the expanded 32-channel first weather image to generate a second feature map.

[0027] Step S24: Use the first feature map and the second feature map as outputs of the dual-channel convolution.

[0028] It should be noted that the convolution step size of the convolution layer is 1, and the convolution step size of the pooling convolution layer is 1, and then the dual channels are fused by the fusion method. The calculation formula is as follows: F = H × W × (C1 + C2); where H and W represent the height and width of the image, and C1 and C2 are the number of channels of the image. When performing dual-channel fusion, ensure that C1 = C2 in the channel.

[0029] Different convolution methods are used for fusion of two channels. For channel one (gradually increasing the number of channels), first convolution is used to convert 32 channels into 64 channels, and then the channels are connected through the C2F layer. Then the 64 channels are converted into 128 channels, and then the channels are connected through the C2F layer, thereby increasing 128 to 256, then to 512, and then to 1024. Through this bottom-up approach, more attention can be paid to the changes from local features to global features.

[0030] Different convolution methods are used for fusion of two channels. For channel two (gradually reducing the number of channels), first convolution is used to convert 1024 channels into 512 channels, and then the channels are connected through the C2F layer. Then the 512 channels are converted into 256 channels, and then the channels are connected through the C2F layer, thereby increasing 256 to 128, then to 64, and then to 32. Through this top-down approach, more attention can be paid to the changes from global features to local features.

[0031] Step S106: Fusing the first feature map and the second feature map to obtain a third feature map, and expanding the channels of the third feature map to 1024 channels.

[0032] Specifically, after obtaining the first feature map and the second feature map, it is necessary to fuse the first feature map and the second feature map to obtain a third feature map, and expand the channels of the third feature map to 1024 channels.

[0033] In one implementation, fusing the first feature map and the second feature map to obtain a third feature map includes: fusing the first feature map and the second feature map through a cross-channel pooling operation to obtain the third feature map.

[0034] In one implementation, expanding the channels of the third feature map to 1024 channels includes: expanding the number of channels of the third feature map to 1024 using a 1x1 convolutional layer.

[0035] Specifically, Figure 5 As shown, the first feature map and the second feature map are fused to obtain a third feature map, and the channels of the third feature map are expanded to 1024 channels, including the following steps: Step S31: The output of the dual-channel convolution, that is, the first feature map and the second feature map, are fused through a cross-channel pooling operation to obtain a fused third feature map.

[0036] Step S32: Use a 1x1 convolutional layer to expand the number of channels of the fused third feature map to 1024.

[0037] Step S33: performing batch normalization processing on the expanded 1024-channel third feature map.

[0038] Step S34: Use the ReLU activation function to perform a nonlinear transformation on the normalized 1024-channel third feature map.

[0039] It should be noted that channel expansion: 1x1 convolution is used to expand the fused feature map to 1024 channels. 1x1 convolution can change the number of channels without changing the spatial dimension, increasing the expressive power of the model.

[0040] Batch normalization: Batch normalization of the 1024-channel third feature map helps stabilize the training process and accelerate convergence. The ReLU activation function is used to introduce nonlinearity and enhance the expressiveness of the model.

[0041] Step 108: Process the expanded third feature map through a channel self-attention mechanism to obtain a fourth feature map.

[0042] In one implementation, the expanded third feature map is processed through a channel self-attention mechanism to obtain a fourth feature map, including: calculating an inter-channel similarity matrix between channels of the expanded third feature map of 1024 channels; performing softmax normalization on the inter-channel similarity matrix; performing weighted summation on the third feature map using the normalized similarity matrix; and performing a residual connection between the result of the weighted summation and the third feature map to obtain the fourth feature map with enhanced attention.

[0043] Specifically, Figure 6 As shown, the expanded third feature map is processed by the channel self-attention mechanism to obtain a fourth feature map, including the following steps: Step S41: Calculate the inter-channel similarity matrix of the normalized 1024-channel third feature map between each channel.

[0044] Step S42: performing softmax normalization processing on the similarity matrix between each channel.

[0045] Step S43: using the normalized similarity matrix to perform weighted summation on the third feature map.

[0046] Step S44: Perform a residual connection on the weighted summation result and the third feature map to obtain an attention-enhanced fourth feature map.

[0047] It should be noted that the similarity between each channel in the third feature map of 1024 channels is calculated to form a similarity matrix. The similarity matrix is ​​normalized by softmax and the similarity is converted into weights. The normalized weights are used to perform weighted summation on the third feature map to emphasize important channels. Residual connection: The result of the weighted summation is residually connected to the third feature map to obtain the fourth feature map with enhanced attention. Residual connection helps alleviate the gradient vanishing problem and retains the original information.

[0048] Step S110: using the classify block of the YOLO framework to classify the fourth feature map to obtain a classification result of the first weather image.

[0049] In one implementation, the fourth feature map is classified using the classify block of the YOLO framework to obtain a classification result of the first weather image, including: inputting the fourth feature map into the classify block of the YOLO framework; in the classify block of the YOLO framework, extracting high-level target features of the fourth feature map through multiple convolutional layers and fully connected layers; based on the high-level target features, using a softmax function to calculate the probability distribution of each category; and selecting the category with the highest probability distribution as the classification result of the first weather image.

[0050] Specifically, Figure 7 As shown, the fourth feature map is classified using the classify block of the YOLO framework, including the following steps: Step S51: Input the attention-enhanced fourth feature map into the classify block of the YOLO framework.

[0051] This step is to pass the fourth feature map processed by the channel self-attention mechanism as input to the classification module of the YOLO framework. The attention-enhanced feature map contains important information of the original image and highlights the key features.

[0052] Step S52: extract high-level target features of the fourth feature map through multiple convolutional layers and fully connected layers.

[0053] Extracting high-level features through multiple convolutional and fully connected layers: In the classify block, the fourth feature map will pass through a series of convolutional layers to further extract higher-level features. Convolutional layers are usually followed by fully connected layers, which map the convolutional features to a more abstract feature space. Multiple convolution and fully connected operations can gradually extract more complex and abstract features, which is helpful for the final classification task.

[0054] Step S53: Based on the high-level target features, the probability distribution of each category is calculated using the softmax function.

[0055] Use the softmax function to calculate the probability distribution of each category: After the last fully connected layer, the network outputs a vector of raw scores. The softmax function converts these raw scores into a probability distribution, ensuring that the sum of the probabilities of all categories is 1.

[0056] For each category i, its probability is calculated as follows: , where z_i is the raw score for category i and j goes through all categories.

[0057] S54: Select the category with the highest probability as the classification result.

[0058] After obtaining the probability of each category, the category with the largest probability value is selected as the final classification result. This can be achieved through the argmax function: predicted_class = argmax(P); where P is the probability distribution of the output of the softmax function. The above scheme includes a complete process from feature fusion, channel expansion, attention mechanism to final classification. The embodiment of the present application involves channel fusion, attention mechanism and residual connection, aiming to improve the accuracy and efficiency of weather image classification.

[0059] The weather image classification method provided in the embodiment of the present application is to expand the three channels of the first weather image to 32 channels; perform convolution processing on the expanded first weather image using a dual-channel top-down and bottom-up sequential scanning method to obtain a first feature map and a second feature map output by the dual-channel convolution processing; fuse the first feature map and the second feature map to obtain a third feature map, and expand the channels of the third feature map to 1024 channels; process the expanded third feature map through a channel self-attention mechanism to obtain a fourth feature map; classify the fourth feature map using the classify block of the YOLO framework to obtain the classification result of the first weather image, and greatly increase the representation capability of the features by expanding the original three-channel image to 32 channels and then to 1024 channels. This enables the model to capture more details and complex image features, which helps to improve the classification accuracy. Then, a dual-channel (top-down and bottom-up) scanning method is used for convolution processing, which can simultaneously capture local and global information of the image. This multi-scale feature fusion helps the model understand the different hierarchical structures of the image.

[0060] The embodiment of the present application also processes the feature map of 1024 channels through the channel self-attention mechanism, and the model can dynamically focus on important feature channels, improving the expressiveness and efficiency of the model. The classify block of the YOLO framework is used for final classification, which makes full use of the advantages of YOLO in target detection and classification tasks, and can achieve fast and accurate classification. The weather image classification method based on dual-channel multi-scale fusion provided by the present application, although the model uses a large number of channels and complex processing steps, can still maintain a high computational efficiency by using efficient convolution operations and the YOLO framework.

[0061] In one implementation, after the classify block of the YOLO framework is used to classify the fourth feature map to obtain the classification result of the first weather image, it also includes: inputting the second weather image into a pre-trained neural network; in the neural network, calculating the fifth feature map and the sixth feature map corresponding to the second weather image; in the neural network, processing the fifth feature map and the sixth feature map through a channel self-attention mechanism to calculate the seventh feature map; in the neural network, calculating the category probability distribution using the classify block of the YOLO framework; and taking the category with the highest probability as the classification result of the second weather image.

[0062] In the embodiment of the present application, a neural network may be trained, and then a subsequent second weather image may be input into the pre-trained neural network, and the second weather image may be classified by the neural network. Figure 8 As shown, the classification of the second weather image by the neural network includes the following steps: Step S61: inputting the new second weather image data into the trained neural network.

[0063] Step S62: Calculate the fifth feature map F = CNN(x) corresponding to the second weather image.

[0064] Where x is the input second weather image and CNN is a convolutional neural network.

[0065] Step S63: Apply the channel self-attention mechanism and calculate A = Attention(F).

[0066] Step S64: Calculate the enhanced sixth feature map E = F + A × F.

[0067] Step S65: Calculate the category probability distribution P = Softmax(FC(E)) through the classify block of the YOLO framework, where FC is a fully connected layer.

[0068] Step S66: Select the category with the highest probability as the final classification result C = argmax(P).

[0069] In one implementation, before the second weather image is input into the pre-trained neural network, the method further includes: repeating the following process until convergence or reaching a preset number of iterations to initialize the neural network parameters; for each training sample, calculating a predicted value; calculating a loss function based on the predicted value and the neural network parameters; calculating a gradient based on the loss function and the neural network parameters; and updating the neural network parameters based on a learning rate and the gradient.

[0070] Specifically, Fig. 9 As shown, before the second weather image is input into the pre-trained neural network, the process of iteratively training the neural network includes the following steps: Step S71: Initialize the neural network parameters θ.

[0071] Step S72: For each training sample (x_i, y_i), calculate the predicted value .

[0072] Step S73: Calculate the loss function .

[0073] Step S74: Calculate the gradient .

[0074] Step S75: Update parameters .

[0075] Where α is the learning rate.

[0076] Step S76: Repeat steps S62 to S65 until convergence or a preset number of iterations is reached.

[0077] The embodiment of the present application makes innovations in the model framework based on the existing YOLOV8 framework. Different convolution kernels are used to output higher channels from lower channels. This method pays more attention to the local features of the image. As the number of channels increases, the number of convolution kernels increases, thereby focusing on global features from local features. In another channel, a top-down method is used for convolution processing. From multiple convolution kernels to fewer convolution kernels, local features can be focused on.

[0078] The accuracy of cloudy classification of traditional images is low, and it is difficult to distinguish cloudy weather from tornadoes. After adopting the classification processing method adopted in the embodiment of the present invention, the classification effect is greatly improved, thereby improving the classification accuracy of weather images.

[0079] It should be noted that the weather image classification method provided in the embodiment of the present application can be executed by a weather image classification device, or a control module in the weather image classification device for executing the weather image classification method. In the embodiment of the present application, the weather image classification device provided in the embodiment of the present application is described by taking the weather image classification device executing the weather image classification method as an example.

[0080] Fig.10 Schematic diagram of the structure of a weather image classification device according to an embodiment of the present application. Fig.10 As shown, the weather image classification device 1000 includes: a first expansion module 1010 , a first processing module 1020 , a second expansion module 1030 , a second processing module 1040 and a classification module 1050 .

[0081] The first expansion module 1010 is used to expand the 3 channels of the first weather image to 32 channels; the first processing module 1020 is used to perform convolution processing on the expanded first weather image using a dual-channel top-down and bottom-up sequential scanning method to obtain a first feature map and a second feature map output by the dual-channel convolution processing; the second expansion module 1030 is used to fuse the first feature map and the second feature map to obtain a third feature map, and expand the channels of the third feature map to 1024 channels; the second processing module 1040 is used to process the expanded third feature map through a channel self-attention mechanism to obtain a fourth feature map; the classification module 1050 is used to classify the fourth feature map using the classify block of the YOLO framework to obtain a classification result of the first weather image.

[0082] In one implementation, the first expansion module 1010 is used to select samples from adjacent samples of each category label according to the category label of the first weather image to obtain multiple category data samples; perform 32 convolution operations on the first weather image of each category in the category sample data, wherein each convolution operation uses a different convolution kernel and bias item; and obtain 32-channel output results of the 32 convolution operations.

[0083] In one implementation, the first processing module 1020 performs top-down convolution processing on the expanded 32-channel first weather image to generate the first feature map; and performs bottom-up convolution processing on the expanded 32-channel first weather image to generate the second feature map.

[0084] In one implementation, the second expansion module 1030 is used to fuse the first feature map and the second feature map through a cross-channel pooling operation to obtain the third feature map.

[0085] In one implementation, the second expansion module 1030 is used to expand the number of channels of the third feature map to 1024 using a 1x1 convolutional layer.

[0086] In one implementation, the first processing module 1020 is used to perform batch normalization processing on the expanded 1024-channel third feature map; and use a ReLU activation function to perform a nonlinear transformation on the normalized 1024-channel third feature map.

[0087] In one implementation, the second processing module 1040 is used to calculate the inter-channel similarity matrix between each channel of the expanded 1024-channel third feature map; perform softmax normalization on the inter-channel similarity matrix; use the normalized similarity matrix to perform weighted summation on the third feature map; and perform a residual connection between the result of the weighted summation and the third feature map to obtain the fourth feature map with enhanced attention.

[0088] In one implementation, the classification module 1050 is used to input the fourth feature map into the classify block of the YOLO framework; in the classify block of the YOLO framework, high-level target features of the fourth feature map are extracted through multiple convolutional layers and fully connected layers; based on the high-level target features, the probability distribution of each category is calculated using the softmax function; and the category with the highest probability distribution is selected as the classification result of the first weather image.

[0089] In one implementation, the classification module 1050 is also used to input the second weather image into a pre-trained neural network; in the neural network, calculate the fifth feature map corresponding to the second weather image; in the neural network, process the fifth feature map through a channel self-attention mechanism to calculate a sixth feature map; in the neural network, calculate the category probability distribution using the classify block of the YOLO framework; and take the category with the highest probability as the classification result of the second weather image.

[0090] In one implementation, the classification module 1050 is further used to repeat the following process until convergence or a preset number of iterations is reached to initialize the neural network parameters; for each training sample, calculate the predicted value; calculate the loss function based on the predicted value and the neural network parameters; calculate the gradient based on the loss function and the neural network parameters; and update the neural network parameters based on the learning rate and the gradient.

[0091] The weather image classification device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0092] The weather image classification device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0093] The weather image classification device provided in the embodiment of the present application can achieve Figures 1 to 9 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0094] like Fig.11 As shown, an embodiment of the present application further provides an electronic device 1100, including a processor 1101 and a memory 1102, wherein the memory 1102 stores a program or instruction that can be run on the processor 1101, and when the program or instruction is executed by the processor 1101, the following is achieved: expanding the 3 channels of the first weather image to 32 channels; performing convolution processing on the expanded first weather image using a dual-channel top-down and bottom-up sequential scanning method to obtain a first feature map and a second feature map output by the dual-channel convolution processing; fusing the first feature map and the second feature map to obtain a third feature map, and expanding the channels of the third feature map to 1024 channels; processing the expanded third feature map through a channel self-attention mechanism to obtain a fourth feature map; and classifying the fourth feature map using the classify block of the YOLO framework to obtain a classification result of the first weather image.

[0095] In one implementation, based on the category label of the first weather image, sample selection is performed on adjacent samples of each category label to obtain multiple category data samples; 32 convolution operations are performed on the first weather image of each category in the category sample data, wherein each convolution operation uses a different convolution kernel and bias item; and 32-channel output results of the 32 convolution operations are obtained.

[0096] In one implementation, a top-down convolution process is performed on the expanded 32-channel first weather image to generate the first feature map; and a bottom-up convolution process is performed on the expanded 32-channel first weather image to generate a second feature map.

[0097] In one implementation, the first feature map and the second feature map are fused through a cross-channel pooling operation to obtain the third feature map.

[0098] In one implementation, the number of channels of the third feature map is expanded to 1024 using a 1x1 convolutional layer.

[0099] In one implementation, after the number of channels of the third feature map is expanded to 1024 using the 1x1 convolutional layer, batch normalization is performed on the expanded 1024-channel third feature map; and a ReLU activation function is used to perform a nonlinear transformation on the normalized 1024-channel third feature map.

[0100] In one implementation, the inter-channel similarity matrix of the expanded 1024-channel third feature map between each channel is calculated; the inter-channel similarity matrix is ​​subjected to softmax normalization processing; the third feature map is weighted summed using the normalized similarity matrix; the result of the weighted summation is residually connected with the third feature map to obtain the fourth feature map with enhanced attention.

[0101] In one implementation, the fourth feature map is input into the classification block of the YOLO framework; in the classification block of the YOLO framework, high-level target features of the fourth feature map are extracted through multiple convolutional layers and fully connected layers; based on the high-level target features, the probability distribution of each category is calculated using a softmax function; and the category with the highest probability distribution is selected as the classification result of the first weather image.

[0102] In one implementation, after the fourth feature map is classified by using the classify block of the YOLO framework to obtain the classification result of the first weather image, the second weather image is input into a pre-trained neural network; in the neural network, the fifth feature map corresponding to the second weather image is calculated; in the neural network, the fifth feature map is processed by a channel self-attention mechanism to calculate a sixth feature map; in the neural network, the classify block of the YOLO framework is used to calculate the category probability distribution; and the category with the highest probability is used as the classification result of the second weather image.

[0103] In one implementation, before the second weather image is input into the pre-trained neural network, the following process is repeated until convergence or a preset number of iterations is reached to initialize the neural network parameters; for each training sample, a prediction value is calculated; based on the prediction value and the neural network parameters, a loss function is calculated; based on the loss function and the neural network parameters, a gradient is calculated; and based on the learning rate and the gradient, the neural network parameters are updated.

[0104] The specific execution steps can refer to the various steps of the above-mentioned weather image classification method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0105] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals, or other devices except terminals.

[0106] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0107] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0108] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0109] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned weather image classification method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0110] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as ROM, RAM, magnetic disk or optical disk.

[0111] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0113] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A weather image classification method, characterized in that: include: Expand the 3 channels of the first weather image to 32 channels; Performing convolution processing on the expanded first weather image using a dual-channel top-down and bottom-up sequential scanning method to obtain a first feature map and a second feature map output by the dual-channel convolution processing; The first feature map and the second feature map are merged to obtain a third feature map, and the channels of the third feature map are expanded to 1024 channels; Processing the expanded third feature map through a channel self-attention mechanism to obtain a fourth feature map; The fourth feature map is classified by using the classify block of the YOLO framework to obtain a classification result of the first weather image.

2. The method according to claim 1, characterized in that: The step of expanding the 3 channels of the first weather image to 32 channels includes: According to the category label of the first weather image, sample selection is performed on adjacent samples of each category label to obtain multiple category data samples; Performing 32 convolution operations on the first weather image of each category in the category sample data, wherein each convolution operation uses a different convolution kernel and bias item; Get the 32-channel output results of the 32 convolution operations.

3. The method according to claim 1, characterized in that The method of performing convolution processing on the expanded first weather image by using a dual-channel top-down and bottom-up sequential scanning method to obtain a first feature map and a second feature map output by the dual-channel convolution processing includes: Performing top-down convolution processing on the expanded 32-channel first weather image to generate the first feature map; A bottom-up convolution process is performed on the expanded 32-channel first weather image to generate a second feature map.

4. The method according to claim 1, characterized in that The fusing the first feature map and the second feature map to obtain a third feature map includes: The first feature map and the second feature map are fused through a cross-channel pooling operation to obtain the third feature map.

5. The method according to claim 1, characterized in that The step of expanding the channels of the third feature map to 1024 channels comprises: The number of channels of the third feature map is expanded to 1024 using a 1x1 convolutional layer.

6. The method according to claim 5, characterized in that After the number of channels of the third feature map is expanded to 1024 by using the 1x1 convolution layer, the method further includes: Performing batch normalization processing on the expanded third feature map of 1024 channels; The ReLU activation function is used to perform nonlinear transformation on the normalized 1024-channel third feature map.

7. The method according to claim 1, characterized in that The third feature map expanded by the channel self-attention mechanism is processed to obtain a fourth feature map, including: Calculate the inter-channel similarity matrix of the third feature map of the expanded 1024 channels between each channel; Performing softmax normalization processing on the inter-channel similarity matrix; Performing weighted summation on the third feature map using the normalized similarity matrix; The result of the weighted summation is residually connected with the third feature map to obtain the fourth feature map with enhanced attention.

8. The method according to claim 1, characterized in that: The classifying block of the YOLO framework is used to classify the fourth feature map to obtain a classification result of the first weather image, including: Input the fourth feature map into the classification block of the YOLO framework; In the classification block of the YOLO framework, high-level target features of the fourth feature map are extracted through multiple convolutional layers and fully connected layers; Based on the high-level target features, a softmax function is used to calculate the probability distribution of each category; The category with the highest probability distribution is selected as the classification result of the first weather image.

9. The method according to claim 1, characterized in that: After the fourth feature map is classified by using the classify block of the YOLO framework to obtain the classification result of the first weather image, the method further includes: Input the second weather image into the pre-trained neural network; In the neural network, calculating a fifth feature map corresponding to the second weather image; In the neural network, the fifth feature map is processed by a channel self-attention mechanism to calculate a sixth feature map; In the neural network, the classify block of the YOLO framework is used to calculate the category probability distribution; The category with the highest probability is taken as the classification result of the second weather image.

10. The method according to claim 9, characterized in that Before inputting the second weather image into the pre-trained neural network, the method further includes: Repeat the following process until convergence or the preset number of iterations is reached. Initialize neural network parameters; For each training sample, calculate the predicted value; Calculating a loss function based on the predicted value and the neural network parameters; Calculating a gradient according to the loss function and the neural network parameters; The neural network parameters are updated according to the learning rate and the gradient.

Images