Lightning channel segmentation method based on dynamic channel graph convolution and multi-scale attention

By using dynamic channel graph convolution and multi-scale attention methods, the problem of noise and light changes in lightning channel image recognition is solved, and the recognition accuracy and robustness of lightning channel are realized. sex.

CN117115177BActive Publication Date: 2024-12-03NANTONG UNIV +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311059822.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-08-22
Publication Date
2024-12-03
Estimated Expiration
2043-08-22

AI Technical Summary

Technical Problem

The prior art is susceptible to noise and light changes in lightning image recognition, and traditional methods perform poorly in complex scenarios, making it difficult to effectively extract information of lightning channels.

Method used

By adopting the lightning channel segmentation method based on dynamic channel graph convolution and multi-scale attention, the lightning channel segmentation data set, data preprocessing, local detail feature extraction, global dependency modeling, dynamic graph convolution relationship modeling and multi-scale Attention feature map size reduction, combined with deep learning models to achieve accurate identification of lightning channels.

Benefits of technology

Improve the generalization and robustness of lightning channel recognition, can effectively identify lightning channels in various scenarios, improve the recognition ability of lightning channels without obvious lightning channels, and suppress interference from irrelevant information, significantly improve the recognition accuracy and robustness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117115177B_ABST
    Figure CN117115177B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to a lightning channel segmentation method based on dynamic channel graph convolution and multi-scale attention. The present invention introduces a semantic segmentation model into the lightning channel recognition task for the first time, and introduces a baseline network based on the relevant characteristics of the lightning channel image. First, the Transfomer layer is introduced to model the global dependency of local features, which improves the network's recognition ability for inconspicuous lightning channels. Secondly, the dynamic graph convolution is introduced to model channel relationships, which improves the information loss problem of the Transformer layer when encoding sequences. Finally, multi-scale attention is introduced in the jump connection structure, which supplements local detail information while suppressing the interference of irrelevant information. The improved network has a significant improvement in quantitative indicators and qualitative evaluation compared with the baseline network, verifying the effectiveness of the method of the present invention. The method of the present invention is more generalized and robust than the method based on the traditional image segmentation algorithm, and can effectively identify lightning channels in a variety of scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a lightning channel segmentation method based on dynamic channel graph convolution and multi-scale attention. Background Art

[0002] Lightning is one of the major serious natural disasters, which can cause forest and oil depot fires, power supply and communication information system failures or damage, and pose a major threat to aerospace, mining and some important and sensitive high-tech equipment. After the 1980s, the hazards caused by lightning increased significantly, especially in areas closely related to high-tech, such as aerospace, national defense, communications, electricity, computers, and the electronics industry. The widespread use of large-scale and ultra-large-scale integrated circuits that are extremely sensitive to lightning electromagnetic interference has greatly increased the probability of being struck by lightning.

[0003] With the continuous advancement of remote sensing technology, we can obtain more and more abundant lightning image data. However, when analyzing these data, how to automatically extract the information of lightning channels to improve efficiency has become an extremely important task. Usually, the threshold setting method can be used to set a threshold based on image brightness or color features to determine whether a pixel belongs to a lightning channel. When the pixel exceeds the set threshold, the pixel is considered to be part of the lightning channel. Although this method is simple and computationally efficient, it is easily affected by interference factors such as image noise and illumination changes, resulting in unstable recognition results. At the same time, traditional manual recognition, setting global thresholds and local thresholds often perform poorly when processing complex and changeable lightning digital image scenes. "Application of three value methods in lightning channel image recognition" (Journal of Applied Meteorology (2014)), "An adaptive backlighting image processing algorithm based on threshold segmentation" (Computers and Digital Engineering (2020)). Edge detection methods can also be used to process lightning channel images. "Image preprocessing of low-contrast lightning channels" (Electronic World (2021)). This method extracts the boundaries of lightning channels by detecting edge information in the image. Commonly used edge detection algorithms include Sobel, Canny, etc. Edge detection methods can capture the shape and shape of the lightning channel well, but may be sensitive to weak edges or noise.

[0004] Digital lightning images have many characteristics, including that lightning channels are in the shape of thin and long lines with obvious line segment features. The presence of cloud occlusion, raindrops and bifurcation of lightning channels makes it more difficult to extract lightning channels. The weak contrast of some lightning channel bifurcations also increases the difficulty of data analysis. In order to solve these problems, more advanced algorithms and methods are needed, such as semantic segmentation algorithms, which mark each pixel of the image as a lightning channel or a non-lightning channel to achieve pixel-level classification. By training deep learning models, high recognition accuracy and robustness can be achieved. The semantic segmentation algorithm can extract the specific contour and shape information of the lightning channel, which has advantages in lightning channel recognition in complex scenes. Summary of the invention

[0005] The purpose of the present invention is to solve the problems in the prior art and propose a lightning channel segmentation method based on dynamic channel graph convolution and multi-scale attention.

[0006] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0007] The lightning channel segmentation method based on dynamic channel graph convolution and multi-scale attention includes the following steps: S1, lightning channel segmentation dataset construction; S2, data preprocessing; S3, high-resolution local detail feature extraction; S4, global dependency modeling based on self-attention mechanism; S5, global channel relationship modeling based on dynamic graph convolution; S6, feature map size restoration based on multi-scale attention; S7, loss calculation and iterative optimization; S8, training and verification.

[0008] Further, as a preferred technical solution of the present invention, step S1 specifically includes: acquiring a digital image of the lightning channel by a high-speed camera in thunderstorm weather, and selecting a typical digital image of the lightning channel by data cleaning and data de-duplication, the size of the obtained lightning channel image is 256*512, and then the lightning channel in the image is finely annotated pixel by pixel by the labelme annotation software, wherein the background area is annotated with black pixels (0, 0, 0), and the lightning channel area is annotated with white pixels (255, 255, 255) and the annotated image is saved, and finally several groups of lightning channel images and their corresponding annotated images are obtained, and then they are divided into a training set and a test set in a ratio of 7:3 to complete the construction of the lightning channel segmentation data set.

[0009] Further, as a preferred technical solution of the present invention, step S2 specifically includes: performing data enhancement on the data set, and the data enhancement operations adopted include performing denoising, filtering, contrast enhancement, random horizontal flipping and random vertical flipping operations on the image in sequence.

[0010] Further as a preferred technical solution of the present invention, step S3 specifically includes:

[0011] S3.1. Normalize the input lightning channel image X, and the normalized output result x∈R 3×256×512 for:

[0012]

[0013] In formula (1), u is the image mean, δ is the standard deviation of the image;

[0014] S3.2. Use the deep residual network ResNet50 to extract local features of the image, which includes the input block input, the first residual block Resblock1, the second residual block Resblock2, the third residual block Resblock3, and the fourth residual block Resblock4; the output of the input layer input is x 1 ∈R 64×128×256 ;

[0015] The four residual blocks are constructed by connecting multiple residual basic blocks in series. The output of a single residual basic block can be represented as:

[0016] Block(x 1 )=F(x 1 ,{x 1 ,w i})+w i x 1 (2)

[0017] In formula (2), the first term is the mapping output constructed by the basic convolution, and the second term corresponds to the mapping output of the cross-layer convolution. When the residual is 0, the residual basic block is equivalent to an identity mapping. The residual basic block learns new residual features through the above mechanism.

[0018] S3.3, after the output feature map x of the deep residual network ResNet50 5 ∈R 512×32×64 Can be characterized as:

[0019]

[0020] Further, as a preferred technical solution of the present invention, the input block input adopts three convolutional layers conv1_1, conv1_2 and conv1_3, the parameters of the first two convolutional layers conv1_1 and conv1_2 are set as follows: the number of convolutional kernels is 64, the width of the convolutional kernel is 3, the height of the convolutional kernel is 3, the step size is 1, and the padding is 0; the parameters of the third convolutional layer conv1_3 are set as follows: the number of convolutional kernels is 64, the width of the convolutional kernel is 3, the height of the convolutional kernel is 3, the step size is 2, and the padding is 1; each convolutional layer has a built-in activation function, and its type is a Relu function; the first residual block Resblock1, the second residual block Resblock2, the third residual block Resblock3 and the fourth residual block Resblock4 are respectively composed of 3, 4, 6 and 3 residuals The residual basic block is constructed by cascading difference blocks; the single residual basic block structure consists of three basic convolutions conv2_1, conv2_2, conv2_3 and a cross-layer convolution conv2_4; the parameters of each convolution layer are set as follows: the number of convolution kernels for conv2_1 is the number of channels of the input feature map, the convolution kernel width is 1, the convolution kernel height is 1, the step size is 1, and the padding is 0; the number of convolution kernels for conv2_2 is the number of channels of the input feature map, the convolution kernel width is 3, the convolution kernel height is 3, the step size is 1, and the padding is 1; the number of convolution kernels for conv2_3 is 4 times the number of channels of the input feature map, the convolution kernel width is 1, the convolution kernel height is 1, the step size is 1, and the padding is 0; the number of convolution kernels for conv2_4 is 4 times the number of channels of the input feature map, the convolution kernel width is 1, the convolution kernel height is 1, the step size is 1, and the padding is 0.

[0021] Further as a preferred technical solution of the present invention, step S4 specifically includes:

[0022] S4.1. Obtain high-resolution local feature map x 5 After that, the high-resolution local feature map is first divided into multiple fixed-size Patches p∈R by patch embedding. P×P , P is the patch window size, and then the patches are concatenated into sequences

[0023] I p =(Embedding(x 5 )) reshape (4)

[0024] In formula (4), Embedding() represents the patch segmentation operation, () reshape Represents the patch splicing operation, C 1 is the number of input feature map channels, N is the number of split patches, N is 196, P is 16, and the output patch sequence after patch embedding is

[0025] S4.2, Patch sequence is transformed into Patch sequence I through linear projection layer p The dimension is compressed to obtain the input token sequence z that is suitable for the Transformer encoding layer, and the category code is embedded before the token sequence, and the position code is embedded after the token sequence to retain the position information and category information after the image segmentation. The whole process is characterized as follows:

[0026]

[0027] In formula (5), E pos is the embedded position encoding information, x class Encode information for the embedded category, Represents a linear projection operation. The size of the output token sequence after the linear projection layer is z∈R 197×768 , the position encoding is characterized by sine and cosine functions of different frequencies:

[0028]

[0029] In formula (6), pos is the position of the current token, 2i and 2i+1 are the dimensions, the sine function is used when the dimension is even, and the cosine function is used when the dimension is odd. model is the vector dimension in the Transformer model;

[0030] S4.3. A single Transformer layer based on the self-attention mechanism is composed of a multi-head self-attention module MHSA and a multi-layer neural network module MLP, and a residual connection is set before and after the module. The token sequence z first models the long-distance dependency of the local feature sequence through MHSA:

[0031] z l =MHSA(LN(z l-1 ))+z l-1 l=1,2,3...L (7)

[0032] In formula (7), L is the number of Transformer layers, LN() represents the layer normalization operation, and MHSA() represents the multi-head self-attention operation. The core calculation process is as follows:

[0033]

[0034] In formula (8), i represents the self-attention mechanism at the i-th layer, and Q represents the input token z l By i qThe query vector generated after linear transformation, K represents the input token z l By i k The key vector generated after linear transformation, V represents the key vector generated by the input token z l By i v The value vector generated after linear transformation, softmax() means using the softmax function to activate the operation, () T represents the tensor transpose operation, d k is a scaling factor, which is used to prevent the gradient from disappearing due to excessive dot product results after attention weighting. concat() represents feature concatenation operation; w o is the weight between multiple self-attentions, provided by a linear transformation;

[0035] S4.4. After the MHSA module establishes the global dependency, the MLP completes the status update of the global dependency:

[0036] z l ′=MLP(LN(z l ))+z l l=1,2,3...L (9)

[0037] In formula (9), L is the number of Transformer layers, LN() represents the layer normalization operation, MLP() represents the multi-layer neural network calculation operation, and the MLP module is composed of a fully connected layer, a GELU activation layer, and a Dropout layer. Its core calculation process is as follows:

[0038] MLP(z l )=Dropout j (GELU j (Linear j (z l ))) j=0,1 (10)

[0039] In formula (10), j represents the jth MLP operation, Linear() represents the fully connected operation; the number of Transformer layers is 12; S4.5, after completing the state update of global dependencies, according to the idea of ​​linear projection, the output token sequence z l ′ is inversely mapped to feature map x 6 ∈R 512×32×64 .

[0040] Further as a preferred technical solution of the present invention, step S5 specifically includes:

[0041] S5.1. After global average pooling GAP, it is converted into a 512-dimensional column vector w to learn the weight coefficient of each channel; the column vector w is input into the adjacency matrix activation function and then multiplied by a learnable parameter matrix; the adjacency matrix A∈R 512 ×512 Calculated by the following formula:

[0042]

[0043] In formula (10), w i ,w j are the i-th and j-th elements in the column vector w, respectively, T() is the adjacency matrix activation function, Θ ij ∈R 512×512 is a learnable dynamic parameter matrix; T() is a continuous function ranging from 0 to 1, which maps the obtained channel weight vector to a matrix with diagonal elements of 1; the elements other than the diagonal in the adjacency matrix represent the weights of the edges between the corresponding channel nodes, which are expressed by Θ ij Dynamically adjust the connection weights and use this parameter to fine-tune the topological relationship to obtain the adjacency matrix A; the node matrix X is composed of the feature graph x 6 The dimension transformation results in reshaping the features of a single channel into 4096×1 as the features of the corresponding nodes in the graph data; the feature map x 6 A graph data G with 512 nodes is constructed, where each node contains a feature vector with a dimension of 4096×1;

[0044] S5.2. After completing the construction of the node matrix and adjacency matrix, the graph node information aggregation and state update are realized by simplifying the graph convolutional neural network GCN. During the training process, the adjacency matrix learns the edge weights of the node features, which reflects the global relationship of the structured texture features in the dense pixel space. After information diffusion, each node aggregates all the necessary node information and updates its state through linear transformation. The forward propagation law of the dynamic graph convolutional network is defined as:

[0045]

[0046] Among them, H (l) is the output of the lth layer, W is a learnable parameter; δ is an activation function such as the rectified linear unit (ReLU); X is the feature matrix of the nodes in the graph data G; the original adjacency matrix A is Laplace smoothed as

[0047]

[0048] In formula (13), I N is the unit matrix, D = diag (d 1 ,d 2 ,…,dn ) is A+I N The degree matrix of .

[0049] Further, as a preferred technical solution of the present invention, step S6 specifically includes: after completing the dynamic channel graph convolution operation, the feature graph x 7 ∈R 512×32×64 The channel dimension and spatial dimension are restored through the cascade upsampling module; the feature map x is obtained after the first upsampling module 8 ∈R 256×64×128 , and then the local feature map x 3 ∈R 256×64×128 After the multi-scale attention module strengthens the important local detail features and suppresses irrelevant noise, it is then used as a skip connection with the feature map x 8 ∈R 256 ×64×128 splicing in the channel dimension; the multi-scale attention module consists of two point convolution layers, ordinary convolution layers and dilated convolution layers, batch normalization layers and softmax functions; the feature map x 3 , the calculation process is as follows:

[0050]

[0051] In formula (14), Pointconv() represents a point convolution layer, where the outer point convolution layer is used to restore the channel dimension, the inner point convolution layer is used for feature extraction, Ordinaryconv() represents an ordinary convolution layer, and Dilationconv() represents an expansion convolution layer, which extracts multi-scale information of local features with different receptive fields through convolution kernels of different sizes;

[0052] Through the cascaded upsampling module and multi-scale module, the channel and spatial dimensions of the input feature map are finally restored to obtain the predicted lightning channel segmentation map.

[0053] Further, as an optimal technical solution of the present invention, there are 4 cascade upsampling modules, and a single cascade upsampling module is composed of an upsampling operator, a 3×3 convolution layer and a RELU activation layer, and the upsampling operator is implemented by bilinear interpolation; the parameters of the inner point convolution layer are set as follows: the number of convolution kernels is the number of input feature map channels, the convolution kernel width is 1, the convolution kernel height is 1, the step size is 1, and the padding is 0; the number of convolution kernels of the outer point convolution layer is three times the number of input feature map channels; the parameters of the ordinary convolution are set as follows: the number of convolution kernels is the number of input feature map channels, the convolution kernel width is 3, the convolution kernel height is 3, the step size is 2, and the padding is 1; the parameters of the dilated convolution are set as follows: the number of convolution kernels is the number of input feature map channels, the convolution kernel width is 5, the convolution kernel height is 5, the step size is 4, and the padding is 3.

[0054] Further, as a preferred technical solution of the present invention, the segmentation loss is calculated by segmenting the prediction map and the segmentation annotation map, and then an alternating parameter update strategy is used to alternately update the relevant weight coefficients in the forward and backward propagation to complete the overall optimization of the model, thereby completing the training; the loss calculation method is shown in formula (14):

[0055] Loss = λ 1 ζ ce (S,S gt )+λ 2 ζ dice (E,E gt ) (15)

[0056] In formula (15), ζ ce is the cross entropy loss, ζ dice is the dice loss, λ 1 ,λ 2 They are the weight coefficients of segmentation loss and edge perception loss, respectively, and their values ​​are both 0.5.

[0057] The lightning channel segmentation method based on dynamic channel graph convolution and multi-scale attention described in the present invention adopts the above technical solution and has the following technical effects compared with the prior art:

[0058] (1) Compared with the methods based on traditional image segmentation algorithms, this method is more generalizable and robust and can effectively identify lightning channels in various scenarios.

[0059] (2) This method firstly introduces the semantic segmentation model into the lightning channel recognition task, and introduces the baseline network according to the relevant characteristics of the lightning channel image. Firstly, the Transformer layer is introduced to model the global dependency of local features, which improves the network's recognition ability of inconspicuous lightning channels. Secondly, the dynamic graph convolution is introduced to model the channel relationship, which improves the information loss problem of the Transformer layer when encoding the sequence. Finally, multi-scale attention is introduced in the skip connection structure, which supplements the local detail information while suppressing the interference of irrelevant information. The improved network has a significant improvement in quantitative indicators and qualitative evaluation compared with the baseline network, which verifies the effectiveness of this method. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of a method flow of an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of a model structure of an embodiment of the present invention;

[0062] Figure 3(A) to Figure 3(C) A schematic diagram of the development process of natural lightning in different time periods. DETAILED DESCRIPTION

[0063] The present invention is further explained below in detail with reference to the accompanying drawings so that those skilled in the art can more deeply understand the present invention and be able to implement it. However, the following reference examples are only used to explain the present invention and are not intended to limit the present invention.

[0064] like Figure 1 As shown in the figure, a lightning channel segmentation method based on dynamic channel graph convolution and multi-scale attention includes the following steps: S1, lightning channel segmentation dataset construction; in thunderstorm weather, lightning channel digital images are obtained by collecting with a high-speed camera, and typical lightning channel digital images are selected by data cleaning and data de-reselection, and the obtained lightning channel images are all of size 256*512, and then the lightning channels in the images are finely annotated pixel by pixel through labelme annotation software, wherein the background area is annotated with black pixels (0, 0, 0), and the lightning channel area is annotated with white pixels (255, 255, 255), and the annotated images are saved, and finally 20 groups of lightning channel images and their corresponding annotated images are obtained, and then they are divided into training set and test set in a ratio of 7:3, and the construction of the lightning channel segmentation dataset is completed.

[0065] S2. Data preprocessing: Data enhancement is performed on the data set. The data enhancement operations used include: denoising, filtering, contrast enhancement, random horizontal flipping, and random vertical flipping of the image in sequence. Data enhancement operations can increase the diversity of data, making the final trained model have better generalization performance.

[0066] S3, high-resolution local detail feature extraction: First, the input lightning channel image X is normalized, and the normalized output result x∈R 3×256×512 for:

[0067]

[0068] In formula (1), u is the image mean and δ is the standard deviation of the image. Then, the deep residual network ResNet50 is used to extract the local features of the image, which includes the input block input, the first residual block Resblock1, the second residual block Resblock2, the third residual block Resblock3, and the fourth residual block Resblock4. The output of the input layer input is x 1 ∈R 64×128×256In this embodiment, the input block input uses three convolutional layers conv1_1, conv1_2 and conv1_3. The parameters of the first two convolutional layers conv1_1 and conv1_2 are set as follows: the number of convolutional kernels is 64, the width of the convolutional kernel is 3, the height of the convolutional kernel is 3, the step size is 1, and the padding is 0. The parameters of the third convolutional layer conv1_3 are set as follows: the number of convolutional kernels is 64, the width of the convolutional kernel is 3, the height of the convolutional kernel is 3, the step size is 2, and the padding is 1. Each convolutional layer has a built-in activation function, and its type is the Relu function.

[0069] The four residual blocks are constructed by connecting multiple residual basic blocks in series. In this embodiment, the first residual block Resblock1, the second residual block Resblock2, the third residual block Resblock3 and the fourth residual block Resblock4 are constructed by connecting 3, 4, 6 and 3 residual basic blocks in cascade, respectively. The structure of a single residual basic block consists of three basic convolutions conv2_1, conv2_2, conv2_3 and a cross-layer convolution conv2_4. In this implementation, the parameters of each convolution layer are set as follows: conv2_1: the number of convolution kernels is the number of channels of the input feature map, the convolution kernel width is 1, the convolution kernel height is 1, the step length is 1, and the padding is 0; conv2_2: the number of convolution kernels is the number of channels of the input feature map, the convolution kernel width is 3, the convolution kernel height is 3, the step length is 1, and the padding is 1; conv2_3: the number of convolution kernels is 4 times the number of channels of the input feature map, the convolution kernel width is 1, the convolution kernel height is 1, the step length is 1, and the padding is 0; conv2_4: the number of convolution kernels is 4 times the number of channels of the input feature map, the convolution kernel width is 1, the convolution kernel height is 1, the step length is 1, and the padding is 0. The output of a single residual basic block can be represented as:

[0070] Block(x 1 )=F(x 1 ,{x 1 ,w i})+w i x 1 (2)

[0071] In formula (2), the first term is the mapping output constructed by the basic convolution, and the second term corresponds to the mapping output of the cross-layer convolution. When the residual is 0, the residual basic block is equivalent to an identity mapping. The residual basic block can learn new residual features through the above mechanism to alleviate the performance degradation problem of the network.

[0072] After the output feature map x of the deep residual network ResNet50 5 ∈R 512×32×64 Can be characterized as:

[0073]

[0074] S4. Global dependency modeling based on self-attention mechanism; obtain high-resolution local feature map x 5 After that, the high-resolution local feature map is first divided into multiple fixed-size Patches p∈R by patch embedding. P×P , P is the patch window size, and then the patches are concatenated into sequences

[0075] I p =(Embedding(x 5 )) reshape (4)

[0076] In formula (4), Embedding() represents the patch segmentation operation, () reshape Represents the patch splicing operation, C 1 is the number of input feature map channels, N is the number of split patches. In this embodiment, N is 196, P is 16, and the output patch sequence after patching is

[0077] The Patch sequence is transformed into Patch sequence I through the linear projection layer p The dimension is compressed to obtain the input token sequence z that is suitable for the Transformer encoding layer, and the category code is embedded before the token sequence and the position code is embedded after the token sequence to retain the position information and category information after the image segmentation. The whole process is characterized as follows:

[0078]

[0079] In formula (5), E pos is the embedded position encoding information, x class Encode information for the embedded category, Represents a linear projection operation. In this embodiment, the size of the output token sequence after the linear projection layer is z∈R 197×768 , the position encoding is characterized by sine and cosine functions of different frequencies:

[0080]

[0081] In formula (6), pos is the position of the current token, 2i and 2i+1 are the dimensions, the sine function is used when the dimension is even, and the cosine function is used when the dimension is odd. model It is the vector dimension in the Transformer model.

[0082] A single Transformer layer based on the self-attention mechanism is composed of a multi-head self-attention module (MHSA) and a multi-layer neural network module (MLP) alternating with each other, and a residual connection is set before and after the module. The token sequence z is first modeled by MHSA to model the long-distance dependency of the local feature sequence:

[0083] z l =MHSA(LN(z l-1 ))+z l-1 l=1,2,3...L (7)

[0084] In formula (7), L is the number of Transformer layers, LN() represents the layer normalization operation, and MHSA() represents the multi-head self-attention operation. The core calculation process is as follows:

[0085]

[0086] In formula (8), i represents the self-attention mechanism at the i-th layer, and Q represents the input token z l By i q The query vector generated after linear transformation, K represents the input token z l By i k The key vector generated after linear transformation, V represents the key vector generated by the input token z l By i v The value vector generated after linear transformation, softmax() means using the softmax function to activate the operation, () T represents the tensor transpose operation, d k is a scaling factor, which is used to prevent the gradient from disappearing due to excessive dot product results after attention weighting. concat() represents the feature concatenation operation. o is the weight between multiple self-attentions, provided by a linear transformation.

[0087] After the MHSA module establishes global dependencies, the MLP completes the status update of global dependencies:

[0088] z l ′=MLP(LN(z l ))+z l l=1,2,3...L (9)

[0089] In formula (9), L is the number of Transformer layers, LN() represents the layer normalization operation, MLP() represents the multi-layer neural network calculation operation, and the MLP module is composed of a fully connected layer, a GELU activation layer, and a Dropout layer. Its core calculation process is as follows:

[0090] MLP(z l )=Dropout j (GELU j (Linear j (z l )))j=0,1 (10)

[0091] In formula (10), j represents the jth MLP operation, and Linear() represents the fully connected operation. In this embodiment, the number of Transformer layers is 12. After completing the state update of global dependencies, the output token sequence z is transformed into l ′ is inversely mapped to feature map x 6 ∈R 512×32×64 .

[0092] S5. Global channel relationship modeling based on dynamic graph convolution; In order to fully capture the dynamic topological information between feature channels and effectively highlight the features in a single channel, the feature graph x 6 The global channel relationship is modeled through dynamic graph convolution. First, it is converted into a 512-dimensional column vector w after global average pooling (GAP) to learn the weight coefficient of each channel. In order to obtain the topological correlation between channels, the column vector w is input into the adjacency matrix activation function and then multiplied by a learnable parameter matrix to ensure the dynamic performance of the adjacency matrix. The adjacency matrix A∈R 512×512 It can be calculated by the following formula:

[0093]

[0094] In formula (10), w i ,w j are the i-th and j-th elements in the column vector w, respectively, T() is the adjacency matrix activation function, Θ ij ∈R 512×512 is a learnable dynamic parameter matrix. T() is a continuous function ranging from 0 to 1, which maps the obtained channel weight vector to a matrix with diagonal elements of 1. The elements in the adjacency matrix other than the diagonal represent the weights of the edges between the corresponding channel nodes, which are expressed by Θ ij Dynamically adjust the connection weights and use this parameter to fine-tune the topological relationship to obtain the adjacency matrix A. The node matrix X is composed of the feature graph x 6The dimension transformation results in reshaping the features of a single channel into 4096×1 as the features of the corresponding nodes in the graph data. 6 A graph data G with 512 nodes has been constructed, where each node contains a feature vector of dimension 4096×1.

[0095] After completing the construction of the node matrix and adjacency matrix, the graph node information aggregation and state update are realized by simplifying the graph convolutional neural network (GCN). During the training process, the adjacency matrix learns the edge weights of the node features, which reflects the global relationship of the structured texture features in the dense pixel space. After the information is diffused, each node aggregates all the necessary node information and updates its state through linear transformation. The forward propagation law of the dynamic graph convolutional network is defined as:

[0096]

[0097] Among them, H (l) is the output of the lth layer, W is a learnable parameter. δ is an activation function such as the rectified linear unit (ReLU). X is the feature matrix of the nodes in the graph data G. In order to alleviate the problem of gradient disappearance or explosion, the original adjacency matrix A is Laplace smoothed as

[0098]

[0099] Where I N is the unit matrix, D = diag (d 1 ,d 2 ,…,d n ) is A+I N The degree matrix of .

[0100] S6, feature map size restoration based on multi-scale attention; after completing the dynamic channel graph convolution operation, the feature map x 7 ∈R 512×32×64 The channel dimension and spatial dimension are restored through the cascade upsampling module. In this embodiment, there are 4 cascade upsampling modules. A single cascade upsampling module consists of an upsampling operator, a 3×3 convolution layer and a RELU activation layer. The upsampling operator is implemented by bilinear interpolation. After the first upsampling module, the feature map x is obtained. 8 ∈R 256×64×128 , and then the local feature map x 3 ∈R 256×64×128 After the multi-scale attention module strengthens the important local detail features and suppresses irrelevant noise, it is then used as a skip connection with the feature map x 8 ∈R 256×64×128The multi-scale attention module consists of two point convolution layers, ordinary convolution layers and dilated convolution layers, batch normalization layers and softmax functions. 3 For example, the calculation process is as follows:

[0101]

[0102] In formula (14), Pointconv() represents a point convolution layer, in which the outer point convolution layer is used for channel dimension restoration, the inner point convolution layer is used for feature extraction, Ordinaryconv() represents an ordinary convolution layer, and Dilationconv() represents an expansion convolution layer, which extracts multi-scale information of local features with different receptive fields through convolution kernels of different sizes. In this embodiment, the parameters of the inner point convolution layer are set as follows: the number of convolution kernels is the number of channels of the input feature map, the convolution kernel width is 1, the convolution kernel height is 1, the step length is 1, and the padding is 0. The number of convolution kernels of the outer point convolution layer is three times the number of channels of the input feature map; the parameters of the ordinary convolution are set as follows: the number of convolution kernels is the number of channels of the input feature map, the convolution kernel width is 3, the convolution kernel height is 3, the step length is 2, and the padding is 1; the parameters of the expansion convolution are set as follows: the number of convolution kernels is the number of channels of the input feature map, the convolution kernel width is 5, the convolution kernel height is 5, the step length is 4, and the padding is 3.

[0103] Through the cascaded upsampling module and multi-scale module, the channel and spatial dimensions of the input feature map are finally restored to obtain the predicted lightning channel segmentation map.

[0104] S7, loss calculation and iterative optimization: The segmentation loss is calculated by segmenting the prediction map and the segmentation annotation map, and then the alternating parameter update strategy is used to alternately update the relevant weight coefficients in the forward and backward propagation to complete the overall optimization of the model, thereby completing the training. The loss calculation method is shown in formula (14).

[0105] Loss = λ 1 ζ ce (S,S gt )+λ 2 ζ dice (E,E gt ) (15)

[0106] In formula (15), ζ ce is the cross entropy loss, ζ dice is the dice loss, λ 1 ,λ 2 are weight coefficients of segmentation loss and edge perception loss respectively. In this embodiment, their values ​​are both 0.5.

[0107] S8, training and validation; Figure 2As shown in the figure, after the model training is completed, the optimized segmentation model realizes the pixel-by-pixel segmentation of the given lightning channel test image. And the evaluation index is calculated to verify the effect of this method. The selected evaluation index is the intersection over union (IOU) of the lightning channel area. The baseline model of the experiment uses the ResUnet semantic segmentation network, whose IOU index is 65.55%. After the introduction of the Transformer structure, the IOU index increases to 67.19%. On this basis, after the introduction of multi-scale attention, the IOU index increases to 68.62%. After the introduction of dynamic channel graph convolution, the IOU index increases to 69.77%, which is about 4.22% higher than the IOU index of the ResUnet network, proving the effectiveness of this method.

[0108] like Figure 3(A)-Figure 3(C) As shown in the figure, the segmentation results of the original image (a) and the improved image (b): (a) is a high-speed camera system Phantom V711 produced by Vision Research Company to shoot the lightning development process with high time resolution. The shooting frame rate of the V711 camera is 14000 frames per second, the resolution is 1280×720 pixels, and the exposure time of a single frame image is 20μs. The pictures show the development process of the same natural lightning in different time periods. Figure 3(A)-Figure 3(C) (b) shows the semantic segmentation of the present invention based on the original image in (a), and it can be seen that relatively efficient lightning channel recognition is achieved.

[0109] The specific implementation scheme described above further describes in detail the purpose, technical scheme and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims

1. A lightning channel segmentation method based on dynamic channel graph convolution and multi-scale attention, characterized by: The following steps are involved: S1, lightning channel segmentation dataset construction; S2, data preprocessing; S3, high-resolution local detail feature extraction; S4, global dependency modeling based on self-attention mechanism; S5. Global channel relationship modeling based on dynamic graph convolution; S6, Feature map size restoration based on multi-scale attention; S7, loss calculation and iterative optimization; S8, training and validation; Step S1 specifically includes: acquiring a lightning channel digital image by a high-speed camera in thunderstorm weather, and selecting a typical lightning channel digital image by data cleaning and data de-reselection, wherein the size of the obtained lightning channel image is 256*512, and then the lightning channel in the image is finely annotated pixel by pixel by the labelme annotation software, wherein the background area is annotated with black pixels (0, 0, 0), and the lightning channel area is annotated with white pixels (255, 255, 255), and the annotated image is saved, and finally several groups of lightning channel images and their corresponding annotated images are obtained, and then they are divided into a training set and a test set in a ratio of 7:3, and the construction of the lightning channel segmentation data set is completed; Step S2 specifically includes: performing data enhancement on the data set, wherein the data enhancement operations used include performing denoising, filtering, contrast enhancement, random horizontal flipping, and random vertical flipping operations on the image in sequence; Step S3 specifically includes: S3.

1. Normalize the input lightning channel image X, and the normalized output result x∈R 3×256×512 for: In formula (1), u is the image mean, δ is the standard deviation of the image; S3.

2. Use the deep residual network ResNet50 to extract local features of the image, which includes the input block input, the first residual block Resblock1, the second residual block Resblock2, the third residual block Resblock3, and the fourth residual block Resblock4; the output of the input layer input is x1∈R 64×128×256 ; The four residual blocks are constructed by connecting multiple residual basic blocks in series. The output of a single residual basic block can be represented as: Block(x1)=F(x1,{x1,w i })+w i x1 (2) In formula (2), the first term is the mapping output constructed by the basic convolution, and the second term corresponds to the mapping output of the cross-layer convolution. When the residual is 0, the residual basic block is equivalent to an identity mapping. The residual basic block learns new residual features through the above mechanism. S3.3, after the output feature map x5∈R of the deep residual network ResNet50 512×32×64 Can be characterized as: The input block input uses three convolutional layers conv1_1, conv1_2 and conv1_3. The parameters of the first two convolutional layers conv1_1 and conv1_2 are set as follows: the number of convolutional kernels is 64, the width of the convolutional kernel is 3, the height of the convolutional kernel is 3, the step size is 1, and the padding is 0. The parameters of the third convolutional layer conv1_3 are set as follows: the number of convolutional kernels is 64, the width of the convolutional kernel is 3, the height of the convolutional kernel is 3, the step size is 2, and the padding is 1; each convolutional layer has a built-in activation function, and its type is a Relu function; the first residual block Resblock1, the second residual block Resblock2, the third residual block Resblock3 and the fourth residual block Resblock4 are constructed by cascading 3, 4, 6 and 3 residual basic blocks respectively. ; The single residual basic block structure consists of three basic convolutions conv2_1, conv2_2, conv2_3 and a cross-layer convolution conv2_4; The parameters of each convolution layer are set as follows: the number of convolution kernels for conv2_1 is the number of channels of the input feature map, the convolution kernel width is 1, the convolution kernel height is 1, the step length is 1, and the padding is 0; the number of convolution kernels for conv2_2 is the number of channels of the input feature map, the convolution kernel width is 3, the convolution kernel height is 3, the step length is 1, and the padding is 1; the number of convolution kernels for conv2_3 is 4 times the number of channels of the input feature map, the convolution kernel width is 1, the convolution kernel height is 1, the step length is 1, and the padding is 0; the number of convolution kernels for conv2_4 is 4 times the number of channels of the input feature map, the convolution kernel width is 1, the convolution kernel height is 1, the step length is 1, and the padding is 0; Step S4 specifically includes: S4.

1. After obtaining the high-resolution local feature map x5, the high-resolution local feature map is first divided into multiple fixed-size Patches p∈R by patch embedding. P×P , P is the patch window size, and then the patches are concatenated into sequences I p =(Embedding(x5)) reshape (4) In formula (4), Embedding() represents the patch segmentation operation, () reshape represents the patch concatenation operation, C1 is the number of input feature map channels, N is the number of split patches, N is 196, P is 16, and the output patch sequence after patch embedding is S4.2, Patch sequence is transformed into Patch sequence I through linear projection layer p The dimension is compressed to obtain the input token sequence z that is suitable for the Transformer encoding layer, and the category code is embedded before the token sequence, and the position code is embedded after the token sequence to retain the position information and category information after the image segmentation. The whole process is characterized as follows: In formula (5), E pos is the embedded position encoding information, x class Encode information for the embedded category, Represents a linear projection operation. The size of the output token sequence after the linear projection layer is z∈R 197×768 , the position encoding is characterized by sine and cosine functions of different frequencies: In formula (6), pos is the position of the current token, 2i and 2i+1 are the dimensions, the sine function is used when the dimension is even, and the cosine function is used when the dimension is odd. model is the vector dimension in the Transformer model; S4.

3. A single Transformer layer based on the self-attention mechanism is composed of a multi-head self-attention module MHSA and a multi-layer neural network module MLP, and a residual connection is set before and after the module. The token sequence z first models the long-distance dependency of the local feature sequence through MHSA: z l =MHSA(LN(z l-1 ))+z l-1 l=1,2,3...L (7) In formula (7), L is the number of Transformer layers, LN() represents the layer normalization operation, and MHSA() represents the multi-head self-attention operation. The core calculation process is as follows: In formula (8), i represents the self-attention mechanism at the i-th layer, and Q represents the input token z l By i q The query vector generated after linear transformation, K represents the input token z l By i k The key vector generated after linear transformation, V represents the key vector generated by the input token z l By i v The value vector generated after linear transformation, softmax() means using the softmax function to activate the operation, () T represents the tensor transpose operation, d k is a scaling factor, which is used to prevent the gradient from disappearing due to excessive dot product results after attention weighting. concat() represents feature concatenation operation; w o is the weight between multiple self-attentions, provided by a linear transformation; S4.

4. After the MHSA module establishes the global dependency, the MLP completes the status update of the global dependency: With l ′=MLP(LN(z l ))+z l l=1,2,3...L (9) In formula (9), L is the number of Transformer layers, LN() represents the layer normalization operation, MLP() represents the multi-layer neural network calculation operation, and the MLP module is composed of a fully connected layer, a GELU activation layer, and a Dropout layer. Its core calculation process is as follows: MLP(from l )=Dropout j (GEL j (Linear j (from l ))) j=0.1 (10) In formula (10), j represents the jth MLP operation, Linear() represents the fully connected operation; the number of Transformer layers is 12; S4.

5. After completing the global dependent state update, according to the idea of ​​linear projection, the output token sequence z l ′ is inversely mapped to feature map x6∈R 512×32×64 ; Step S5 specifically includes: S5.

1. After global average pooling GAP, it is converted into a 512-dimensional column vector w to learn the weight coefficient of each channel; the column vector w is input into the adjacency matrix activation function and then multiplied by a learnable parameter matrix; the adjacency matrix A∈R 512×512 Calculated by the following formula: In formula (10), w i ,w j are the i-th and j-th elements in the column vector w, respectively, T() is the adjacency matrix activation function, Θ ij ∈R 512×512 is a learnable dynamic parameter matrix; T() is a continuous function ranging from 0 to 1, which maps the obtained channel weight vector to a matrix with diagonal elements of 1; the elements other than the diagonal in the adjacency matrix represent the weights of the edges between the corresponding channel nodes, which are expressed by Θ ij The connection weights are adjusted dynamically, and the topological relationship is fine-tuned using this parameter to obtain the adjacency matrix A. The node matrix X is obtained by transforming the feature graph x6 dimensions, and the features of a single channel are reshaped into 4096×1 as the features of the corresponding nodes in the graph data. The feature graph x6 constructs a graph data G with 512 nodes, where each node contains a feature vector with a dimension of 4096×1. S5.

2. After completing the construction of the node matrix and adjacency matrix, the graph node information aggregation and state update are realized by simplifying the graph convolutional neural network GCN. During the training process, the adjacency matrix learns the edge weights of the node features, which reflects the global relationship of the structured texture features in the dense pixel space. After information diffusion, each node aggregates all the necessary node information and updates its state through linear transformation. The forward propagation law of the dynamic graph convolutional network is defined as: Among them, W is a learnable parameter; δ is the rectified linear unit ReLU activation function; X is the feature matrix of the node in the graph data G; Laplace smoothing of the original adjacency matrix A is In formula (13), I N is the unit matrix, D = diag(d1, d2, …, d n ) is A+I N The degree matrix of Step S6 specifically includes: after completing the dynamic channel graph convolution operation, the feature map x7∈R 512×32×64 The channel dimension and spatial dimension are restored through the cascade upsampling module; after the first upsampling module, the feature map x8∈R is obtained 256 ×64×128 , and then the local feature map x3∈R 256×64×128 After the multi-scale attention module strengthens the important local detail features and suppresses irrelevant noise, it is then used as a skip connection with the feature map x8∈R 256×64×128 Splicing in the channel dimension; the multi-scale attention module consists of two point convolution layers, ordinary convolution layers and dilated convolution layers, batch normalization layers and softmax functions; the feature map x3, its processing and calculation process is as follows: In formula (14), Pointconv() represents a point convolution layer, where the outer point convolution layer is used to restore the channel dimension, the inner point convolution layer is used for feature extraction, Ordinaryconv() represents an ordinary convolution layer, and Dilationconv() represents an expansion convolution layer, which extracts multi-scale information of local features with different receptive fields through convolution kernels of different sizes; The channels and spatial dimensions of the input feature map are finally restored through the cascaded upsampling module and the multi-scale module to obtain the predicted lightning channel segmentation map; There are 4 cascade upsampling modules in total. A single cascade upsampling module consists of an upsampling operator, a 3×3 convolution layer and a RELU activation layer. The upsampling operator is implemented by bilinear interpolation. The parameters of the inner point convolution layer are set as follows: the number of convolution kernels is the number of input feature map channels, the convolution kernel width is 1, the convolution kernel height is 1, the step length is 1, and the padding is 0. The number of convolution kernels in the outer point convolution layer is three times the number of input feature map channels. The parameters of the ordinary convolution are set as follows: the number of convolution kernels is the number of input feature map channels, the convolution kernel width is 3, the convolution kernel height is 3, the step length is 2, and the padding is 1. The parameters of the dilated convolution are set as follows: the number of convolution kernels is the number of input feature map channels, the convolution kernel width is 5, the convolution kernel height is 5, the step length is 4, and the padding is 3. The segmentation loss is calculated by segmenting the prediction map and the segmentation annotation map, and then the alternating parameter update strategy is used to alternately update the relevant weight coefficients in the forward and backward propagation to complete the overall optimization of the model and thus complete the training; the loss calculation method is shown in formula (14): Loss=λ1ζ ce (S,S gt )+λ2ζ dice (E,E gt ) (15) In formula (15), ζ ce is the cross entropy loss, ζ dice is the dice loss, λ1 and λ2 are the weight coefficients of segmentation loss and edge perception loss, respectively, and their values ​​are both 0.5.

Citation Information

Patent Citations

  • Method for predicting thunder and lightning

    CN113204903A

  • Semantic segmentation method for multi-scale context information interaction based on attention guidance

    CN115115831A