Mesoscale convective system recognition and tracking method based on image anchor-free frame detection

Through a deep convolutional neural network without anchor frame detection, the category and border distance are predicted by pixel point, and the mask Mask is generated, which solves the problems of low MCS recognition efficiency and high computational cost in traditional methods, and achieves more efficient identification and tracking Effect.

CN112836713BActive Publication Date: 2024-11-05NANJING UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202110270336.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-03-12
Publication Date
2024-11-05
Estimated Expiration
2041-03-12

AI Technical Summary

Technical Problem

The existing mesoscale convective system recognition methods rely on traditional image features and heuristic preset anchor boxes, resulting in low recognition efficiency, large calculation amount, and difficulty in effectively monitoring and forecasting the location and development trend of MCS.

Method used

A deep convolutional neural network based on image anchor-free frame detection is adopted to predict categories and border distances through pixel-by-pixel point full convolution, generate mask Masks, and realize multi-time continuous tracking of MCS.

Benefits of technology

It improves the segmentation performance and tracking accuracy of MCS recognition, reduces network parameters and training inference speed, avoids the problems of anchor box scale and ratio sensitivity, and enhances the modeling ability of MCS geometric deformation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112836713B_ABST
    Figure CN112836713B_ABST
Patent Text Reader

Abstract

The invention discloses a mesoscale convective system (MCS) identification and tracking method based on image anchor-free detection, comprising the following steps: step 1, preprocessing original geostationary satellite infrared brightness temperature data, annotating the mesoscale convective system on the infrared cloud image obtained after the processing, and then randomly dividing the training set, the verification set and the test set; step 2, constructing an instance segmentation network based on the anchor-free network, the network is used to extract image features, detect the mesoscale convective system and segment specific instances; step 3, training set image enhancement, and using transfer learning to supervise the training instance segmentation convolutional neural network to automatically learn network parameters; step 4, using the trained model to detect and segment the mesoscale convective system on the geostationary satellite infrared cloud image at adjacent times; step 5, tracking the mesoscale convective system according to the relevant target matching principle.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and computer vision technology, and in particular relates to a method for identifying and tracking a mesoscale convection system based on image anchor-free frame detection. Background Art

[0002] In recent years, global warming has led to more frequent and active severe convective weather in various places. Mesoscale Convective System (MCS) is a weather system with strong convection, which has the characteristics of short life cycle and small spatial scale. It is often accompanied by catastrophic weather such as heavy rain, typhoon, hail, etc., which endangers people's lives and causes huge losses to the national economy. Especially in civil aviation operations, the strong convective movement of MCS can cause severe turbulence of the aircraft, and even cause serious damage to the fuselage, leading to air crashes. Therefore, how to quickly and accurately identify MCS instances and analyze their evolution and movement is crucial, which is an important research topic in today's meteorological discipline. In the past, MCS identification and tracking was mainly based on conventional observation data. However, these data cover a small area and are affected by various factors and have large errors. Therefore, it is difficult to effectively monitor and predict the location and development trend of MCS. With the improvement of satellite remote sensing and radar detection capabilities, especially high temporal and spatial resolution geostationary satellites are widely used in MCS monitoring applications due to their strong reliability, continuous observation, and high accuracy.

[0003] At present, most MCS recognition methods are based on traditional image features, that is, they are identified according to relevant judgment criteria. These methods are too dependent on the selection of feature thresholds and the entire process is cumbersome and computationally intensive. With the rapid development of deep learning, especially deep convolutional neural networks, complex computer vision tasks such as target detection, instance segmentation, and target tracking have made breakthrough progress. Therefore, how to use deep convolutional neural networks to quickly, simply and effectively segment MCS instances is of great significance for subsequent MCS tracking and related meteorological analysis. Summary of the invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is that the traditional identification methods of mesoscale convective systems in the meteorological field have various judgment criteria, and a mesoscale convective system identification and tracking method based on image anchor-free detection is proposed. Considering that the morphological scale of MCS varies greatly, and the current mainstream instance segmentation method based on heuristic pre-set anchor boxes is not ideal in this application, a pixel-by-pixel full convolution method is used to directly predict the category to which each position in the Feature Map belongs, the distance to the four sides of the corresponding border, and the production mask Mask, and on this basis, the MCS is effectively tracked continuously for multiple times.

[0005] The method of the present invention specifically comprises the following steps:

[0006] Step 1: preprocess the original infrared brightness temperature data file according to the relevant instructions of the satellite data: including cropping of the original satellite data, marking of image polygon instances, and random division of the training set, validation set, and test set;

[0007] Step 2: Construct a convolutional neural network for instance segmentation of mesoscale convection systems based on anchor-free frames: it is divided into a backbone network for feature extraction, a feature pyramid network that integrates multi-scale features, a prediction frame network head for classification, distance frame regression, and prediction of center-ness, a mask generation network head, and a network head for predicting MaskIoU;

[0008] Step 3: Perform multi-scale image enhancement on the training set, and use transfer learning to supervise the training of the mesoscale convection system instance segmentation convolutional neural network built in step 2 to automatically learn network parameters: use small batch stochastic gradient descent method for training, and set the loss functions of the three network heads: predicting bounding boxes, generating masks, and predicting MaskIoU;

[0009] Step 4, using the trained network to segment the mesoscale convective system instances on the geostationary satellite infrared cloud images at adjacent moments, and obtaining the relevant records of the mesoscale convective system at continuous moments;

[0010] Step 5: Based on step 4, the mesoscale convective system is tracked according to the principle of matching related targets, while taking into account the common splitting and merging situations;

[0011] Step 1 of the present invention comprises:

[0012] The data set used in the present invention is the full-resolution (4km) brightness temperature data (60N-60S) for monitoring global precipitation obtained from the 11-micron infrared channels on the geostationary satellites GMS-5, GOES-8, Goes-10, Meteosat-7, and Meteosat-5 provided by the National Oceanic and Atmospheric Administration (NOAA). In order to reduce the discontinuity between adjacent geostationary satellites, the acquired data has been corrected for vertex correlation, and the data is finally stored in the form of a rectangular grid. The file name format of each infrared brightness temperature data is merg_yyyymmddhh_4km-pixel, where yyyy represents the year (such as 2020), mm represents the month (ranging from 01-12), dd represents the date (ranging from 01-31), and hh represents the hour (ranging from 00-23). Each file contains two records, which are the brightness temperature data at 0 and 30 minutes of the hour. The size of each record is 9896×3298 (the span in the 9896 dimension is 0.036378335 and the span in the 3298 dimension is 0.036383683). At the same time, in order to store data with a size of one byte (8 bits with a range of 0-255), the obtained data is obtained by subtracting 75 from the actual monitored brightness temperature value. The 255 (corresponding to the actual brightness temperature value of 330) in the obtained data is a missing value.

[0013] Step 1-1, read each original geostationary satellite infrared brightness temperature data file according to the relevant instructions of the satellite data, the file is an array in the format of 2×3298×9896, and obtain two grayscale cloud images at the corresponding time 0 and 30 minutes;

[0014] Step 1-2: Since the size of the remote sensing satellite image is 3298×9896, and the image is too large to be suitable for the input of the subsequent instance segmentation network, the grayscale cloud image obtained in step 1-1 is cropped. According to the latitude of the Jianghuai region of my country is 27N to 40N and the longitude is 110E to 125E, combined with the relevant description of the brightness temperature data above, the size of the Jianghuai region of my country is:

[0015]

[0016] Enlarge the above results a little bit, and crop each grayscale cloud image into two or more (usually 240) grayscale infrared cloud sub-images with a size of 420×360;

[0017] Steps 1-3, give polygon instance-level annotations of mesoscale convective systems for each grayscale infrared cloud sub-image, filter out those sub-images without mesoscale convective systems, and obtain the json file corresponding to each grayscale infrared cloud sub-image. The entire annotation has only one category. And the grayscale infrared cloud sub-images are randomly divided into 9407 training set images, 3135 validation set images, and 3136 test set images in a ratio of 6:2:2, thus forming the training set, validation set, and test set.

[0018] Step 2 includes:

[0019] Step 2-1, construct a backbone network for feature extraction: The backbone network in the segmentation network of the present invention uses a convolutional neural network VoVNetV2-99 with a total of 99 layers (reference: An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detection.). The convolutional layers in the convolutional neural network VoVNetV2-99 are all in the form of Conv-BN-ReLU, that is, a combination of a convolutional layer, a batch normalization layer BN, and a linear rectification function ReLU. At the same time, the convolutional neural network VoVNetV2-99 uses a cross-layer connection similar to the residual network ResNet to achieve the identity mapping. The residual block containing the identity mapping is defined as:

[0020] Y=F(X,{W i})+X

[0021] Where X and Y represent the input feature map and output feature map of each structural block respectively, F(X,{W i}) is the residual mapping function that needs to be learned. At the same time, in order to improve the quality of feature representation, the channel attention eSE (Effective Squeeze-Excitation) (reference: CenterMask: Real-Time Anchor-FreeInstance Segmentation.) mechanism is introduced in the network residual block to make the network pay more attention to important feature map channels and suppress irrelevant channels. The specific implementation method is as follows:

[0022] A eSE (X div )=σ(W C (gap(X div )))

[0023]

[0024] Where X divIt is a diversified feature map obtained by cascading network feature maps and reducing their dimensionality. (W and H are the width and height of feature map X respectively) is the channel-level global average pooling, W C is the weight of the fully connected layer, X i,j is the eigenvalue of feature graph X at (i, j), σ represents the Sigmoid activation function, A eSE (X div ) is the feature descriptor of the calculated channel attention, and the feature description is consistent with X div Perform element-wise multiplication (element-wise multiplication is done with the symbol Finally, the refined feature map X is obtained refine .

[0025] The specific steps to build the convolutional neural network VoVNetV2-99 include:

[0026] Step 2-1-1, construct the network stem stage 1: This stage contains three convolution layers: first, a convolution layer with a kernel size of 3×3, a stride of 2, a padding of 1, and an output channel number of 64 (if no additional instructions are given later, the convolution layer with a kernel size of 3×3 defaults to a stride of 1 and a padding of 1) is used to downsample the input image, and then a convolution layer with a kernel size of 3×3 and an output channel number of 64 is connected, and a convolution layer with a kernel size of 3×3, a stride of 2, and an output channel number of 128 is connected. After the input image passes through this stage, the first scale feature map C1 is generated, and the feature map C1 has an Output Stride of 4 compared to the input image scale;

[0027] Step 2-1-2, build the network single-shot aggregation (OSA) module stage 2: This stage contains a residual block, which contains 5 convolution layers with a convolution kernel size of 3×3 and an output channel of 128. After the input image is convolved, the 5 convolution layers each obtain a diversified feature map with a channel number of 128, and on the last convolution layer, the feature map of this layer is cascaded with the feature maps of the first 4 layers and the input first scale feature map C1 to obtain a feature map with 128*6=768 channels, and then the convolution kernel size is 1×1, the step size is 1, the padding is 0, and the output channel number is 256. The convolution layer is used for dimensionality reduction to obtain a diversified feature map X div , combined with the channel attention eSE mechanism to obtain the final X refine , X refineIt is also necessary to perform element-wise addition with the input feature map to achieve identity mapping. Finally, this stage obtains the second-scale feature map C2 with 256 channels, and the scaling ratio of the feature map C2 to the input image is 4.

[0028] Step 2-1-3, build the network single aggregation module stage 3: This stage is similar to the OSA module stage 2, except that this stage first uses a 3×3 maximum pooling layer with a step size of 2 and a padding of 0 to perform a 2x downsampling, and uses a 3×3 adjustable deformable convolution (reference: Deformable ConvNets v2: More Deformable, Better Results.) instead of a conventional convolution in the residual block. The adjustable deformable convolution is defined as:

[0029]

[0030] Where K is the total number of convolution kernel sampling positions (for example, if the convolution kernel size is 3×3, K is 9), w k is the convolution kernel weight, p k is the predefined offset of each k-th position relative to the center of the receptive field (for example, when K is 9, p k ∈{(-1,-1), (-1,0), (-1,1), (0,-1), (0,0), (0,1), (1,-1), (1,0), (1,1)}), x(p) and y(p) are the eigenvalues ​​of the input feature map x at position p and the output feature map y at position p, respectively. Δp k is the learnable offset of the kth position, Δm k ∈[0,1] is the adjustment number of the kth position. The adjustable deformable convolution is implemented by adding an additional convolution layer with the same spatial resolution and expansion rate as the regular convolution to learn the offset and adjustment number of each position of the feature map in the x and y directions of the two-dimensional plane.

[0031] The OSA module stage 3 contains 3 residual blocks, each residual block contains 5 adjustable deformable convolution layers with a convolution kernel size of 3×3 and an output channel of 160. Like the network single aggregation module stage 2 in step 2-1-2, the feature map of this layer is cascaded with the feature maps of the first 4 layers and the input second-scale feature map C2 on the last convolution layer to obtain a diversified feature map with 256+160*5=1056 channels. This is the diversified feature map of the first residual block. Similarly, the number of channels of the diversified feature map obtained after the cascade operation of the second and third residual blocks is 512+160*5=1312, and the number of output channels of the 1×1 conventional convolution layer with a step size of 1 and padding of 0 for dimensionality reduction after the feature map cascade is 512. It also includes a channel attention mechanism. After the OSA module stage 3 operation, the third-scale feature map C3 with a channel number of 512 is finally obtained. The scaling ratio of the feature map C3 to the input image scale is 8;

[0032] Step 2-1-4, construct the network single aggregation module stage 4: This stage is similar to the above OSA module stage 3, except that this stage contains 9 residual blocks, each residual block contains 5 adjustable deformable convolution layers with a convolution kernel size of 3×3 and an output channel of 192. Like the network single aggregation module stage 2 and the network single aggregation module stage 3, the feature map of this layer is cascaded with the feature map of the first 4 layers and the input second-scale feature map C3 on the last convolution layer to obtain a diversified feature map with 512+192*5=1472 channels. This is the diversified feature map of the first residual block. Similarly, the number of channels of the diversified feature map obtained after the cascade operation of the second to ninth residual blocks is 768+192*5=1728), and the number of output channels of the 1×1 conventional convolution layer with a step size of 1 and padding of 0 for dimensionality reduction after the feature map cascade is 768, which also includes the channel attention mechanism. After the network single aggregation module stage 4, the fourth scale feature map C4 is finally obtained. The scaling ratio of the feature map C4 to the input image scale is 16.

[0033] Step 2-1-5, build the last structure of the backbone network, network single aggregation module stage 5: the structure of this stage is similar to stage 4, but this stage contains only 3 residual blocks, each residual block contains 5 adjustable deformable convolution layers with a convolution kernel size of 3×3 and an output channel of 224. Like the network single aggregation module stage 2 and the network single aggregation module stage 3, the feature map of this layer is cascaded with the feature map of the previous 4 layers and the input second-scale feature map C3 on the last convolution layer to obtain a diversified feature map with 768+224*5=1888 channels. This is the diversified feature map of the first residual block. Similarly, the number of channels of the diversified feature map obtained after the cascade operation of the second and third residual blocks is 1024+224*5=2144, and the number of output channels of the 1×1 conventional convolution layer with a step size of 1 and padding of 0 for dimensionality reduction after the feature map cascade is 1024, which also includes the channel attention mechanism. After this stage, the feature map C5 is finally obtained. The scaling ratio of the feature map C5 to the input image is 32.

[0034] Step 2-2, combine the Feature Pyramid Network (FPN) (reference: Feature pyramid networks for object detection.) to fuse multi-scale features. The Feature Pyramid Network (FPN) fuses the features {C3, C4, C5} of different scales obtained in step 2-1 from top to bottom and combines them with lateral connections to obtain {M3, M4, M5}. Among them, M5 is obtained by passing the feature map C5 through a 1×1 convolution layer, and M4 is obtained by element-wise addition of the feature map obtained by performing a 2x nearest neighbor upsampling on M5 and a 1×1 convolution layer on the feature map C4. Similarly, M3 is obtained by performing an element-wise addition of the feature map obtained by performing a 2x nearest neighbor upsampling on M4 and a 1×1 convolution on the feature map C3.

[0035] Finally, each layer in {M3, M4, M5} is subjected to convolution with a kernel size of 3×3 and an output channel number of 256 to reduce the aliasing effect caused by the nearest neighbor interpolation, and the feature layer {P3, P4, P5} is obtained.

[0036] In the instance segmentation network, additional feature layers P6 and P7 are added. The P6 and P7 feature layers are obtained by downsampling P5 and P6 by 2 times through a 3×3 convolution layer with a stride of 2, and finally the feature layers {P3, P4, P5, P6, P7} are obtained;

[0037] Step 2-3, build the instance segmentation bounding box prediction head, which contains two branches, one classification branch, and one multi-task branch parallel to the center-ness (reference: FCOS: Fully Convolutional One-Stage Object Detection.) and distance bounding box regression, including:

[0038] Step 2-3-1, build the classification branch: after each feature layer (i.e. P3-P7) in the feature pyramid network FPN, three regular convolution layers with 256 input and output channels and 3×3 convolution kernel size and an adjustable deformation convolution layer are connected in sequence. These four convolution layers no longer use batch normalization (Batch Normalization, BN) but use group normalization (Group Normalization, GN) to avoid the impact of batch size on the network model. Since only the mesoscale convective system is detected, a prediction classification convolution layer is added at the end of the classification branch, with 256 input channels, 1 output channel, and 3×3 convolution kernel size;

[0039] Step 2-3-2, build a multi-task branch for parallel regression of center and distance border: This part also connects three regular convolutional layers with the same structure as the classification branch and an adjustable deformation convolutional layer in sequence after the feature pyramid network FPN feature layer of each scale. However, after these four convolutional layers, there is a convolutional layer with two parallel branches: a 3×3 convolutional layer with 4 output channels for border regression, and the four dimensional values ​​output by this branch represent the distance from the current position to the four sides of the border; a 3×3 convolutional layer with 1 output channel for predicting center, and the output of this branch is a dimension representing the center value;

[0040] Assume that the distance between a sample point (x, y) on the feature map F and the four edges of the target bounding box to which the point belongs is d = (l, t, r, b), where l, t, r, and b represent the distance from the sample point (x, y) to the left, top, right, and bottom of the rectangular bounding box, respectively. Then the centrality is defined as:

[0041]

[0042] Where min() and max() are the minimum and maximum functions respectively. When using the instance segmentation convolutional neural network for detection, the predicted center value is multiplied by the classification score to get the final confidence. The center branch mainly suppresses low-quality borders that are far from the center of the target object. From the definition of center, it can be found that if the sample point (x, y) is closer to the center of the target border, the center value is closer to 1, otherwise the center value is closer to 0. Therefore, the classification score of distant points multiplied by a smaller center value will get a lower confidence, making it easier to filter the low-quality borders regressed from these distant points in the non-maximum suppression (NMS) stage.

[0043] Step 2-4, similar to the region generation network RPN in anchor-based detectors such as Mask R-CNN, the prediction bounding box head predicts many bounding box regions, which are partially eliminated by non-maximum suppression with an IoU threshold of 0.6 to obtain 100 regions of interest (RoI). Drawing on the implementation principle of Mask R-CNN, the RoI head of the instance segmentation network based on anchor-free boxes is constructed, which specifically includes:

[0044] Step 2-4-1, construct the region of interest alignment RoI Align (reference: Mask R-CNN.) layer: The RoIAlign layer first maps the region of interest RoI to the corresponding feature pyramid network FPN feature layer P according to the size of the region of interest RoI k The implementation is:

[0045] k=Ceil(k max -log2(A input / A RoI ))

[0046] Among them A input With A RoI Represent the area of ​​the input image and the area of ​​the RoI respectively, k max The last layer of the backbone network is set to 5, Ceil() is the rounding function, and k is the number of feature pyramid network FPN feature layers mapped by RoI;

[0047] The RoI Align layer divides the corresponding position area on the mapped feature map into 14×14=196 small areas of equal size, adaptively samples these areas, takes the center point position of each part, and then uses bilinear interpolation to calculate, finally obtaining a 14×14 fixed-size feature map;

[0048] Step 2-4-2, construct the Mask head with spatial attention mechanism: The Mask head contains four consecutive convolution layers with a convolution kernel size of 3×3 and input and output channels of 256. In order to make this branch pay more attention to meaningful pixels, the spatial attention module is introduced after the fourth convolution layer, and its implementation mechanism is as follows:

[0049]

[0050]

[0051] Where X i is the input feature map, A sag (X i )Spatial attention feature descriptor, P max With P avg They represent the maximum pooled feature map and the average pooled feature map in the channel dimension respectively. represents the cascade operation, F 3×3 is a 3×3 convolutional layer, σ is a Sigmoid function, is element-wise multiplication, X sag The feature map that is finally combined with spatial attention;

[0052] Then the resulting X sag The deconvolution layer with a convolution kernel size of 2×2 and a stride of 2 is used for upsampling to obtain a feature map with a size of 28×28 and the same number of channels. The last convolution layer of the Mask head is a Mask prediction layer specific to each category. Since the detection target is the category of mesoscale convective system, the convolution kernel size of the Mask prediction layer is 1×1, the stride is 1, and the number of output channels is 1;

[0053] Step 2-4-3, construct the MaskIoU head, and use the mask score Mask Scoring (reference: MaskScoring R-CNN.) to re-represent the Mask quality: Based on step 2-4-2, the output feature map of the Mask prediction layer is downsampled by 2 times using a 2×2 maximum pooling layer and then cascaded with the RoI feature map of size 14×14 and 256 channels output by the RoI Align layer to obtain a feature map of size 28×28 and 257 channels.

[0054] The MaskIoU head contains four consecutive convolutional layers with a kernel size of 3×3 and an output channel number of 256, and the stride of the last convolutional layer is 2. It is then connected to two fully connected layers with an output channel number of 1024 and one with an output channel number of 1;

[0055] Step 3 of the present invention comprises:

[0056] Step 3-1, since the infrared cloud image in the dataset is a single-channel grayscale image, in order to perform subsequent transfer learning, the grayscale images in the training set are converted into RGB (red, green, and blue) three-channel images. The values ​​of the three RGB channels are the same, which are all grayscale values ​​of the grayscale image. Data enhancement is performed on the converted images in the training set and the corresponding instance annotations: first, the image is scaled according to multiple scales, where the long side is 1333 and the short side is a random one of {640, 672, 704, 736, 768, 800}, and the original image ratio is maintained. Random horizontal flipping is also used for data enhancement.

[0057] The pixel values ​​are centered. Since the training uses a pre-trained model on the ImageNet dataset, the image needs to be normalized according to the mean in the ImageNet dataset. The normalization operation is performed on a channel basis. The mean values ​​of the R, G, and B channels are 123.675, 116.28, and 103.53, respectively, and the standard deviations are 58.395, 57.12, and 57.375, respectively. The scalar combination of the mean values ​​of the three RGB channels is represented as a vector The standard deviation scalar combination of the three RGB channels is represented as a vector σ. Let the input image be x, then the normalized image data x′ is

[0058]

[0059] Then the image scale is padded to a multiple of 32 to avoid feature loss in subsequent convolution operations;

[0060] Accordingly, after the input image is scaled and horizontally flipped for data enhancement, the corresponding instance annotations must also undergo the same transformation to obtain the correct annotations;

[0061] Step 3-2, set the classification loss function for predicting the bounding box head in the instance segmentation network: The classification task uses the focal loss function to improve the class imbalance problem in the one-stage detector. Since the dataset contains only one type of label, only one binary classifier needs to be trained. Considering that the network treats each position as a training sample rather than an anchor box, let the feature map F i (i=3,4,...7) The position (x i ,y i ) The predicted value is set up

[0062]

[0063] in Indicates whether it is a positive or negative sample, Indicates the position (x i ,y i ) is a positive sample, Then it means the position (x i ,y i ) is a negative sample, is the position (x i ,y i ) is the probability of a positive sample;

[0064] The position (x i ,y i ) is mapped to the corresponding position (x', y') of the input image as follows:

[0065]

[0066] where s i is the feature map F i Compared to the Output Stride, which is scaled by the input image scale, if (x', y') falls into any Ground Truth (real mark, GT) border, it is a positive sample with Y = 1, otherwise Y = 0. Focus loss function based on α balance It is expressed as:

[0067]

[0068] Where α is the weight factor, γ≥0 is an adjustable focusing parameter, and α and γ are generally set to 0.25 and 2.0 respectively. i , its total classification loss function for:

[0069]

[0070] Step 3-3, set the centrality loss function: given the feature map F i The positive sample feature point (x i ,y i ) distance bounding box regression target in They represent the feature points (x i ,y i ) to the left, top, right, and bottom of the real bounding box. Then the feature point (x i ,y i )’s centrality target Defined as:

[0071]

[0072] Apparently Therefore, the centrality branch adopts the binary cross entropy loss function during training. Let the feature graph F i After the centrality branch, we get the position (x i ,yi ) Then the feature map F i The total center loss function for:

[0073]

[0074] in Indicates whether it is a positive or negative sample, Indicates the position (x i ,y i ) is a positive sample, Then it means the position (x i ,y i ) are negative samples. For indicator functions, if the condition in parentheses If it holds, the indicator function value is 1, otherwise its value is 0;

[0075] Step 3-4, set the distance regression loss function: The regression task uses the GIoU loss function, set the distance in They represent the feature points (x i ,y i ) to the left, top, right, and bottom of the predicted bounding box. in They represent the feature points (x i ,y i ) to the left, top, right, and bottom of the real border. i ,y i ) falls into multiple GT borders, the border with the smallest area is selected as the distance regression target. and Considered as an external bounding box, the GIoU loss function is expressed as:

[0076]

[0077]

[0078] Where C is and The bounding boxes of the two, || represents the area of ​​the region, Indicates that area C does not contain and Some areas, for and The intersection ratio of for and The generalized intersection-and-union ratio of .

[0079] Feature map F i The total distance regression loss function is:

[0080]

[0081] in The meaning is the same as in step 3-3. is the element-level weighting coefficient of the loss function, which is the value of the position (x i ,y i )Centerness regression target;

[0082] Step 3-5, set the Mask loss function: predict the bounding box head before passing through the Mask Head to obtain multiple proposed bounding box proposals, and obtain a maximum of 100 RoIs for each image based on the score threshold of 0.05 and the non-maximum suppression of the IoU threshold of 0.6. In order to improve the convergence rate and detection performance, the GT bounding box is also added to the RoI for network training. Suppose there are N RoIs with GT bounding boxes added, and K GT bounding boxes. Calculate the IoU between RoI and GT, and divide RoI into positive and negative samples according to the IoU threshold of 0.5, where the positive sample label is 1 and the negative sample label is 0. Obtain the dictionary D corresponding to the i∈[0,N)th RoI and its most matching j∈[0,K)th GT bounding box related information. Finally, sample the training sample X for Mask Head with the ratio of positive samples to 1 / 4 of all samples.

[0083] Since Mask Loss is only defined on positive samples, it is also necessary to perform foreground screening on the training sample X. Suppose that the predicted bounding box head has a total of M positive sample RoIs, and the i-th RoI i After the region of interest is aligned RoIAlign, a feature map F of size 14×14 is obtained i , F i After masking the header Mask Head, we get a 28×28 size prediction feature map pred i Mask target gt_mask i The acquisition of GT is first to find the GT information (including category, border, polygon Mask) according to D through the RoI number, and then cut out the border area on the original image Mask according to the border information, and finally resize the cropped part to 28×28 to get the final gt_mask i The average binary cross entropy loss function is used to calculate L mask :

[0084]

[0085] in and is the feature map pred i and gt_mask i The eigenvalue at the (x,y) position;

[0086] Step 3-6, set the MaskIoU loss function: let the predicted value of the ith feature map after the cascade operation after the predicted mask intersection and comparison head MaskIoU Head be pred_maskiou i , the MaskIoU target is gt_maskiou i In steps 3-5, this value can be obtained based on the predicted Mask and the Mask information in the corresponding GT to obtain gt_maskiou i .use The loss function is used to calculate L maskiou :

[0087]

[0088] Step 3-7, set the multi-task loss function: add the classification loss function, center loss function, distance regression loss function, Mask loss function and MaskIoU loss function to obtain the total multi-task loss function L of the small batch:

[0089]

[0090] Where N cls 、N ctn 、N reg 、N mask 、N maskIoU is the normalization coefficient of the corresponding loss function, and N cls =N ctm =N reg is the number of positive samples in the predicted bounding box head, N mask =N maskIoU The number of positive samples obtained after the proposed bounding box proposals are predicted in the prediction bounding box head according to the IoU threshold and the ratio of positive and negative samples. ctn , cls , cls , cls is the balance coefficient of each loss, and its value is 1;

[0091] Step 3-8: Set network learning related parameters and perform training.

[0092] Steps 3-8 include: using the pre-trained weights of VovNetV2-99 on ImageNet for transfer learning. Select the 1000 proposals with the highest confidence based on the pre-frame detection results, and retain 100 proposals after non-maximum suppression, where the IoU threshold in non-maximum suppression NMS is 0.6. In Mask Head, proposals with an IoU value greater than 0.5 with the true marked box are regarded as positive samples, otherwise they are negative samples, and the number of positive samples used for training accounts for 1 / 4 of all training samples. At the same time, network training uses small batch stochastic gradient descent to optimize the network model, with a batch size of 2 and a learning rate of 0.0025. The Warmup strategy is used in the first 1000 iterations, and a momentum coefficient of 0.9 is used. In terms of regularization, 10 -4 The weight decay coefficient. The entire training epoch is 24 rounds. In the 16th and 22nd rounds, the learning rate step reduction strategy is adopted with a reduction coefficient of 0.1. After setting the above learning parameters, use the training set processed in step 3-1 to train the constructed network;

[0093] Step 4 of the present invention comprises:

[0094] Step 4-1, the image that needs instance segmentation is processed in the same way as in step 3-1, except that the image is only scaled to a single scale during inference, that is, the long side is 1333 and the short side is 800;

[0095] Step 4-2, the trained network performs forward calculation on the test image obtained in step 1-3, removes the proposed bounding box proposals with a confidence less than 0.05 in the predicted bounding box head, and screens out the 1000 proposed bounding box proposals with the highest confidence. A maximum of 50 RoIs are obtained for Mask branch prediction through non-maximum suppression NMS, and finally the kth Mask is obtained according to the category k predicted by the classification branch (but the data used in this article only contains one category). The 28×28 size Mask will be scaled to the corresponding RoI scale and binarized with a threshold of 0.5 to generate the final mask Mask;

[0096] In step 4-3, the top-50 proposals also use the MaskIoU header to predict the MaskIoU, which will be multiplied by the classification confidence to get the Mask confidence.

[0097] Step 4-4, according to the Mask generated in step 4-2 and the Mask confidence obtained in step 4-3, NMS is used with an IoU threshold of 0.5 to filter out the final predicted Mask;

[0098] Step 4-5, segment the mesoscale convective system instances on the geostationary satellite infrared cloud images of the Jianghuai region of my country at adjacent moments, and obtain the mesoscale convective system instances at continuous moments.

[0099] Step 5 of the present invention comprises:

[0100] Step 5-1, find the centroid coordinates of each mesoscale convective system instance Characteristic area, intensity P:

[0101]

[0102] area=N

[0103]

[0104] Where N is the total number of pixels contained in the mesoscale convective system instance, x i With y i is the horizontal and vertical coordinates of the i-th pixel, f(i) is the brightness temperature of the i-th pixel, f z It is the accumulated sum of the lighting temperatures of each pixel in the example. The image resolution needs to be considered in subsequent calculations.

[0105] Step 5-2, calculate the duration, centroid position change, area change and intensity change of the mesoscale convective system at two adjacent moments according to the time sequence; if two mesoscale convective systems at adjacent moments meet the centroid position change ≤50m / s and area change ≤5km 2 / s, and the intensity change is ≤0.001℃ / s, it is preliminarily judged to be the same target;

[0106] Step 5-3, considering the common splitting and merging phenomena of mesoscale convective systems, corresponding processing is required. If there are n mesoscale convective systems MCS at a certain moment, denoted by X1, X2,…,X n and the previous MCS is denoted as P j If the tracking matching principle in step 5-2 is met, then the phenomenon belongs to the splitting situation, and the X with the largest area after splitting is selected. m Continuation P j The trajectory of X, 1≤m≤n; update X m The index value of the previous moment of the action is j and the duration is P j The duration of the new cloud is +1, while other MCSs are regarded as the index value of the previous moment of the new cloud map is empty and the duration is initialized to 1;

[0107] If there are more than two MCSs at the previous moment, denoted as P1, P2, ..., P z And an MCS at that time is denoted as P aIf the MCS tracking matching principle is met, the phenomenon belongs to the merging situation, and the P with the largest area at the previous moment is selected. m (1≤m≤z) as P a The last moment trajectory, update P m The index value of the previous moment of the action is m and the duration is P m The duration of the MCS with the non-largest area at the previous moment ends, and its duration is determined to be ≥ 1h. If so, the corresponding MCS behavior is obtained in reverse order according to the index value at the previous moment, thereby realizing MCS tracking.

[0108] Beneficial effects: MCS identification and tracking are the core concerns of meteorological disaster forecasting. In the past, MCS identification methods were usually based on traditional image features. Such methods rely more on the selection of judgment thresholds, and the whole process involves more image processing technologies, which is relatively cumbersome. The present invention adopts a method based on a deep convolutional neural network to identify MCS. It avoids the problems of sensitivity to parameters such as anchor frame scale, aspect ratio, and number in the detection method based on heuristic pre-set anchor frames through anchor-free full convolution. At the same time, the present invention also combines deformable convolution to further improve the modeling ability of MCS geometric deformation, as well as the attention mechanism to pay more attention to important channels or position information. Compared with other deep learning segmentation methods, the present invention not only achieves better segmentation performance in MCS identification, but also has fewer network parameters and faster training and inference speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and other advantages of the present invention will become more clear.

[0110] Figure 1 is a flow chart of the mesoscale convective system identification and tracking process of the present invention;

[0111] Figure 2 is an original geostationary satellite infrared brightness temperature grayscale image in an embodiment of the present invention;

[0112] Figure 3 The structure of the anchor-free instance segmentation network of the present invention;

[0113] Figure 4 The residual block structure of the backbone network of the example segmentation network of the present invention;

[0114] Figure 5 This is a segmentation effect diagram of a mesoscale convective system example of the present invention;

[0115] Figure 6 This is a diagram showing the effect of tracking the mesoscale convective system of the present invention; DETAILED DESCRIPTION

[0116] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0117] like Figure 1 As shown, the workflow of the mesoscale convective system identification and tracking constructed by the method of the present invention can be roughly divided into four stages: the first stage, the preprocessing and annotation of the original satellite data; the second stage, the construction of the instance segmentation network model; the third stage, the training and inference of the network model; the fourth stage, the tracking of the continuous mesoscale convective system instances detected. The specific construction steps of the method for identifying and tracking mesoscale convective systems in the embodiment of the present invention are as follows:

[0118] Step 1: Since mesoscale convective systems mostly occur in summer, the geostationary satellite infrared brightness temperature data from June to September 2017 from 2000 to 2017 provided by the National Oceanic and Atmospheric Administration of the United States are partially randomly screened as the operating data of the embodiment of the present invention. In addition, the screened raw satellite data needs to be preprocessed:

[0119] (1) According to the relevant instructions of the satellite data, read each original geostationary satellite infrared brightness temperature data file as an array in the format of 2×3298×9896 to obtain the corresponding time 0 minute and 30 minute. Figure 2 In the grayscale cloud image shown, the blank part with a pixel value of 255 is regarded as a missing value;

[0120] (2) Since the size of the remote sensing satellite image is 3298×9896, and the image is too large to be suitable for the input of the subsequent instance segmentation network, the grayscale cloud map obtained in step 1-1 is cropped. According to the latitude of the Jianghuai region of my country is 27N to 40N and the longitude is 110E to 125E, combined with the relevant description of the brightness temperature data above, the size of the Jianghuai region of my country is:

[0121]

[0122] Enlarge the above result a little more and crop the entire grayscale cloud image into multiple 420×360 sub-images;

[0123] (3) Give the polygon instance-level annotation of the mesoscale convective system for each grayscale infrared cloud sub-image, filter out those sub-images without mesoscale convective systems, and obtain the json file corresponding to each image. The entire annotation has only one category. And randomly divide the above sub-images into 9407 training set images, 3135 validation set images, and 3136 test set images in a ratio of 6:2:2;

[0124] Step 2: Construct a convolutional neural network for instance segmentation of mesoscale convective systems based on anchor-free frames. The instance segmentation network structure is as follows: Figure 3 As shown, Figure 3 It includes: a backbone network for extracting deep abstract features of images, which has five stage feature maps, C1, C2, C3, C4, and C5; a feature pyramid network that integrates multi-scale features, which also has five feature layers of different scales, P3, P4, P5, P6, and P7; a network head for predicting borders, which includes two large branches: a classification branch and a large branch parallel to the center-Ness branch and the distance border regression branch. The head is shared by feature pyramid layers of different scales; a network head for predicting masks, which includes a spatial attention mechanism SAM, and before entering the head, the region of interest alignment RoIAlign needs to be performed; a network head for predicting MaskIoU, the input feature map is obtained by concatenating the feature map output by the mask head with the feature map obtained by RoIAlign after Maxpooling downsampling;

[0125] Step 2-1, construct a backbone network for feature extraction. The backbone network in the segmentation network of the present invention uses a convolutional neural network VoVNetV2-99 with a total of 99 layers. The convolutional layers in the backbone network are all in the form of Conv-BN-ReLU, that is, a combination of a convolutional layer, a batch normalization layer BN, and a linear rectification function ReLU. At the same time, the network structure block uses a cross-layer connection similar to that in the residual network ResNet to achieve the identity mapping. The residual block containing the identity mapping is defined as:

[0126] Y=F(X,{W i})+X

[0127] Where X and Y represent the input and output feature maps of each structural block, respectively, F(X,{W i}) is the residual mapping function that needs to be learned. At the same time, in order to improve the quality of feature representation, the channel attention eSE (Effective Squeeze-Excitation) mechanism is introduced in the network residual block to make the network pay more attention to important feature map channels and suppress irrelevant channels. The specific implementation method is as follows:

[0128] A eSE (X div )=σ(W C (gap(X div )))

[0129]

[0130] Where X div It is a diversified feature map obtained by cascading network feature maps and reducing their dimensionality. (W and H are the width and height of feature map X) is the channel-level global average pooling, W C is the weight of the fully connected layer, X i,j is the eigenvalue of feature graph X at (i, j), σ represents the Sigmoid activation function, A eSE (X div ) is the calculated channel attention feature descriptor, which is consistent with X div Perform element-wise multiplication Finally, the refined feature map X is obtained refine , which combines the above identity mapping and the residual block structure of the channel attention mechanism as follows Figure 4 The structure of the entire backbone network VoVNetV2-99 is as follows:

[0131] a. Stem stage 1: This stage contains three convolutional layers: first, a convolutional layer with a kernel size of 3×3, a stride of 2, a padding of 1, and an output channel number of 64 (if no additional instructions are given later, the convolutional layer with a kernel size of 3×3 defaults to a stride of 1 and a padding of 1) is used to downsample the input image, and then a convolutional layer with a kernel size of 3×3 and an output channel number of 64 and a convolutional layer with a kernel size of 3×3, a stride of 2, and an output channel number of 128 are connected. After the input image passes through this stage, the first scale feature map C1 is generated, and the feature map C1 has an Output Stride of 4 compared to the input image scale;

[0132] b. Single-Shot Aggregation (OSA) module stage 2: This stage contains a residual block, which contains 5 convolution layers with a convolution kernel size of 3×3 and an output channel of 128. After the input image is convolved, each of the five convolution layers obtains a diversified feature map with a channel number of 128. In the last convolution layer, the feature map of this layer is cascaded with the feature maps of the first four layers and the input first scale feature map C1 to obtain a feature map with 128*6=768 channels. Then, a convolution layer with a convolution kernel size of 1×1, a step size of 1, a padding of 0, and an output channel number of 256 is used to reduce the dimension and obtain a diversified feature map X. div , combined with the above channel attention mechanism eSE to get the final X refine , X refineIt is also necessary to perform element-wise addition with the input feature map to achieve identity mapping. Finally, the second-scale feature map C2 is obtained at this stage. The scaling ratio of the feature map C2 to the input image is still 4.

[0133] c. OSA module stage 3: This stage is similar to OSA module stage 2, except that this stage first uses a 3×3 max pooling layer with a stride of 2 and padding of 0 to perform 2x downsampling, and uses a 3×3 adjustable deformable convolution instead of a conventional convolution in the residual block. The adjustable deformable convolution can be defined as:

[0134]

[0135] Where K is the total number of convolution kernel sampling positions (for example, if the convolution kernel size is 3×3, K is 9), w k is the convolution kernel weight, p k is the predefined offset of each k-th position relative to the center of the receptive field (for example, when K is 9, p k ∈{(-1,-1), (-1,0), (-1,1), (0,-1), (0,0), (0,1), (1,-1), (1,0), (1,1)}), x(p) and y(p) are the feature values ​​of the input feature map x and the output feature map y at position p, respectively, Δp k is the learnable offset of the kth position, Δm k ∈[0,1] is the adjustment number of the kth position. The adjustable deformable convolution is implemented by adding an additional convolution layer with the same spatial resolution and expansion rate as the regular convolution to learn the offset and adjustment number of each position of the feature map in the x and y directions of the two-dimensional plane.

[0136] OSA module stage 3 contains 3 residual blocks, each residual block contains 5 adjustable deformable convolution layers with a convolution kernel size of 3×3 and an output channel of 160. Like OSA module stage 2 in step 2-1-2, the feature map of this layer is cascaded with the feature maps of the first 4 layers and the input second-scale feature map C2 on the last convolution layer to obtain a diversified feature map with 256+160*5=1056 channels (this is the diversified feature map of the first residual block. Similarly, the number of channels of the diversified feature map obtained after the cascade operation of the second and third residual blocks is 512+160*5=1312), and the number of output channels of the 1×1 conventional convolution layer with a step size of 1 and padding of 0 for dimensionality reduction after the feature map cascade is 512, which also includes the channel attention mechanism. After this stage of operation, the third-scale feature map C3 is finally obtained. The scaling ratio of the feature map C3 to the input image scale is 8;

[0137] d. OSA module stage 4: This stage is similar to the above OSA module stage 3, except that this stage contains 9 residual blocks, each residual block contains 5 adjustable deformable convolution layers with a convolution kernel size of 3×3 and an output channel of 192. Like the OSA module stages 2 and 3, the feature map of this layer is cascaded with the feature maps of the previous 4 layers and the input second-scale feature map C3 on the last convolution layer to obtain a diversified feature map with 512+192*5=1472 channels (this is the diversified feature map of the first residual block. Similarly, the number of channels of the diversified feature map obtained after the cascade operation of the second to ninth residual blocks is 768+192*5=1728), and the number of output channels of the 1×1 conventional convolution layer with a step size of 1 and padding of 0 for dimensionality reduction after the feature map cascade is 768, which also includes the channel attention mechanism. After these convolutional layers, the fourth-scale feature map C4 is finally obtained. The scaling ratio of the feature map C4 to the input image is 16.

[0138] e. OSA module stage 5: The structure of this stage is similar to stage 4, but this stage only contains 3 residual blocks, each residual block contains 5 adjustable deformable convolution layers with a convolution kernel size of 3×3 and an output channel of 224. Like OSA module stages 2 and 3, the feature map of this layer is cascaded with the feature maps of the first 4 layers and the input second-scale feature map C3 on the last convolution layer to obtain a diversified feature map with 768+224*5=1888 channels (this is the diversified feature map of the first residual block. Similarly, the number of channels of the diversified feature map obtained after the cascade operation of the second and third residual blocks is 1024+224*5=2144), and the number of output channels of the 1×1 conventional convolution layer with a step size of 1 and padding of 0 for dimensionality reduction after the feature map cascade is 1024, and the channel attention mechanism is also included. After this stage, the feature map C5 is finally obtained, and the scale ratio of the feature map C5 to the input image is 32;

[0139] Step 2-2, combine the Feature Pyramid Network (FPN) to fuse multi-scale features. FPN fuses the features {C3, C4, C5} of different scales in step 2-1 from top to bottom and combines them with lateral connections to obtain {M3, M4, M5}. M5 is obtained by passing the feature map C5 through a 1×1 convolution layer, M4 is obtained by adding the feature map obtained by performing a 2x nearest neighbor upsampling on M5 and a 1×1 convolution layer on the feature map C4, and similarly, M3 is obtained by adding the feature map obtained by performing a 2x nearest neighbor upsampling on M4 and a 1×1 convolution on the feature map C3. Finally, each layer in {M3, M4, M5} is subjected to a convolution with a kernel size of 3×3 and an output channel number of 256 to reduce the aliasing effect caused by the nearest neighbor interpolation, and obtain the feature layer {P3, P4, P5}. In this example segmentation network, additional feature layers P6 and P7 are added, which are obtained by downsampling P5 and P6 by 2 times through a 3×3 convolutional layer with a stride of 2, and finally obtain the feature layers {P3, P4, P5, P6, P7};

[0140] Step 2-3: Build the instance segmentation bounding box prediction head, which contains two branches: a classification branch and a multi-task branch that performs parallel center-ness and distance bounding box regression:

[0141] Step 2-3-1, build the classification branch: after each feature layer in FPN (i.e. P3-P7), three regular convolution layers with 256 input and output channels and 3×3 convolution kernel size and an adjustable deformation convolution layer are connected in sequence. These four convolution layers no longer use batch normalization (Batch Normalization, BN) but use group normalization (Group Normalization, GN) to avoid the impact of batch size on the network model. Since only the mesoscale convective system is detected, a prediction classification convolution layer is added at the end of this branch, with 256 input channels, 1 output channel, and 3×3 convolution kernel size;

[0142] Step 2-3-2, build a multi-task branch for parallel regression of center and distance border: This part also connects three regular convolutional layers with the same structure as the classification branch and an adjustable deformation convolutional layer in sequence after the FPN feature layer of each scale. However, after these four convolutional layers, there is a convolutional layer with two parallel branches: a 3×3 convolutional layer with 4 output channels for border regression, and the four dimensional values ​​output by this branch represent the distance from the current position to the four sides of the border; a 3×3 convolutional layer with 1 output channel for predicting center, and the output of this branch is a dimension representing the center value;

[0143] Assume that the distance between a sample point (x, y) on the feature map F and the four edges of the target border to which the point belongs is d = (l, t, r, b), where l, t, r, and b represent the distance to the left, top, right, and bottom of the rectangular border, respectively. Then the centrality is defined as:

[0144]

[0145] Where min() and max() are the minimum and maximum functions respectively. When using the instance segmentation convolutional neural network for detection, the predicted center value is multiplied by the classification score to get the final confidence. The center branch mainly suppresses low-quality borders that are far from the center of the target object. From the definition of center, it can be found that if the sample point (x, y) is closer to the center of the target border, the center value is closer to 1, otherwise the center value is closer to 0. Therefore, the classification score of distant points multiplied by a smaller center value will get a lower confidence, making it easier to filter the low-quality borders regressed from these distant points in the non-maximum suppression (NMS) stage.

[0146] Step 2-4, similar to the region generation network RPN in anchor-based detectors such as Mask R-CNN, the prediction bounding box head predicts many bounding box regions, which are partially eliminated after non-maximum suppression with an IoU threshold of 0.6 to obtain 100 RoIs (Region of Interest). Drawing on the implementation principle of Mask R-CNN, the instance segmentation network RoI head based on anchor-free boxes is constructed, which specifically includes:

[0147] Step 2-4-1, construct the region of interest alignment RoI Align layer: This layer first maps the region of interest RoI to the corresponding FPN feature layer P according to its size k The implementation is:

[0148] k=Ceil(k max -log2(A input / A RoI ))

[0149] Among them A input With A RoI Represent the area of ​​the input image and RoI respectively, k max The number of the last layer of the backbone network is set to 5, Ceil() is the rounding function, and k is the number of FPN feature layers mapped to the RoI;

[0150] Then this layer divides the corresponding position area on the mapped feature map into 14×14=196 small areas of equal size, adaptively samples these areas, takes the center point position of each part, and then uses bilinear interpolation to calculate, and finally obtains a 14×14 fixed-size feature map;

[0151] Step 2-4-2, construct the Mask head including the spatial attention mechanism: This part contains four consecutive convolution layers with a convolution kernel size of 3×3 and input and output channels of 256. In order to make this branch pay more attention to meaningful pixels, a spatial attention module is introduced after the fourth convolution layer. Its implementation mechanism is as follows:

[0152]

[0153]

[0154] Where X i is the input feature map, A sag (X i )Spatial attention feature descriptor, P max With P avg Respectively represent the feature maps of maximum pooling and average pooling in the channel dimension, represents the cascade operation, F 3×3 is a 3×3 convolutional layer, σ is a Sigmoid function, is element-wise multiplication, X sag The feature map that is finally combined with spatial attention;

[0155] Then the resulting X sag The deconvolution layer with a convolution kernel size of 2×2 and a stride of 2 is used to obtain a feature map with a size of 28×28 and the same number of channels. The last convolution layer of the Mask head is a Mask prediction layer specific to each category. Since the detection target is the category of mesoscale convective system, the convolution kernel size of the prediction layer is 1×1, the stride is 1, and the number of output channels is 1;

[0156] Step 2-4-3, construct the MaskIoU header, which uses the mask score Mask Scoring to re-represent the Mask quality: First, based on step 2-4-2, the output feature map of the Mask prediction layer is downsampled by 2 times using a 2×2 maximum pooling layer and then cascaded with the RoI feature map of the RoI Align layer with a size of 14×14 and a number of channels of 256 to obtain a feature map of size 28×28 and 257 channels. The MaskIoU header contains four consecutive convolutional layers with a convolution kernel size of 3×3 and an output channel number of 256, and the stride of the last convolutional layer is 2. Subsequently, two fully connected layers with an output channel number of 1024 and one with an output channel number of 1 are connected;

[0157] Step 3: Training of instance segmentation network model, specifically including the following steps:

[0158] Step 3-1, since the infrared cloud image in the dataset is a single-channel grayscale image, in order to perform subsequent transfer learning, the grayscale image is converted into an RGB three-channel image, and the three channel values ​​are the same as the original grayscale value. Data enhancement is performed on the converted training set images and the corresponding instance annotations: First, the image is scaled according to multiple scales, where the long side is 1333 and the short side is a random one of {640, 672, 704, 736, 768, 800}, and the original image ratio is maintained. Random horizontal flipping is also used for data enhancement.

[0159] The pixel values ​​are centered. Since the training uses a pre-trained model on the ImageNet dataset, the image needs to be normalized according to the mean in the ImageNet dataset. The normalization operation is performed on a channel basis. The mean values ​​of the R, G, and B channels are 123.675, 116.28, and 103.53, respectively, and the standard deviations are 58.395, 57.12, and 57.375. The scalar combination of the mean values ​​of the three RGB channels is represented as a vector The standard deviation scalar combination of the three RGB channels is represented as a vector σ. Suppose the input image is x, then the normalized image data x′ is:

[0160]

[0161] Then the image scale is padded to a multiple of 32 to avoid feature loss in subsequent convolution operations;

[0162] Accordingly, after the input image is scaled and horizontally flipped for data enhancement, the corresponding instance annotations must also undergo the same transformation to obtain the correct annotations;

[0163] Step 3-2, set the classification loss function for predicting the bounding box head in the instance segmentation network: The classification task uses the focal loss function to improve the class imbalance problem in the one-stage detector. Since the dataset contains only one type of label, only one binary classifier needs to be trained. Considering that the network treats each position as a training sample rather than an anchor box, let the feature map F i (i=3,4,...7) The position (x i ,y i ) The predicted value is set up

[0164]

[0165] in Indicates whether it is a positive or negative sample, Indicates the position (x i ,y i ) is a positive sample, Then it means the position (x i ,y i ) is a negative sample, Indicates the position (x i ,y i ) is the probability of a positive sample. i ,y i ) is mapped to the corresponding position of the input image:

[0166]

[0167] where s i is the feature map F i Compared to the input image scale ratio Output Stride, if (x', y') falls into any Ground Truth (real mark, GT) border, it is a positive sample with Y = 1, otherwise Y = 0. The focal loss function based on α balance can be expressed as:

[0168]

[0169] Where α is the weight factor, γ≥0 is an adjustable focusing parameter, and α and γ are set to 0.25 and 2.0 respectively in the experiment. i , the total classification loss function is:

[0170]

[0171] Step 3-3, set the centrality loss function: given the feature map F i The positive sample feature point (x i ,y i ) distance bounding box regression target in Represented as feature point (x i ,y i ) to the left, top, right, and bottom of the real bounding box. Then the feature point (x i ,y i ) is defined as:

[0172]

[0173] Apparently Therefore, the centrality branch adopts the binary cross entropy loss function during training. Let the feature graph F i After the centrality branch, we get the position (x i ,y i ) Then the feature map F i The total center loss function is:

[0174]

[0175] in Indicates whether it is a positive or negative sample, Indicates the position (x i ,y i ) is a positive sample, Then it means the position (x i ,y i ) are negative samples. For indicator functions, if the condition in parentheses If it holds, the indicator function value is 1, otherwise its value is 0;

[0176] Step 3-4, set the distance regression loss function: The regression task uses the GIoU loss function, set the distance in Represented as feature point (x i ,y i ) to the left, top, right, and bottom of the predicted bounding box. Regression distance target in Represented as feature point (x i ,y i ) to the left, top, right, and bottom of the real border. i ,y i ) falls into multiple GT borders, the border with the smallest area is selected as the distance regression target. and Considered as an external bounding box, the GIoU loss function can be expressed as:

[0177]

[0178]

[0179] Where C is and The bounding boxes of the two, || represents the area of ​​the region, Indicates that area C does not contain and Some areas, for and The intersection ratio of for and The generalized intersection-over-union ratio of . Feature map F i The total distance regression loss function is:

[0180]

[0181] in The meaning is the same as in step 3-3. is the element-level weighting coefficient of the loss function, which is the value of the position (x i ,y i )Centerness regression target;

[0182] Step 3-5, set the Mask loss function: predict the bounding box head before passing through the Mask Head to obtain multiple proposed bounding box proposals, and obtain a maximum of 100 RoIs for each image based on the score threshold of 0.05 and the non-maximum suppression of the IoU threshold of 0.6. In order to improve the convergence rate and detection performance, the GT bounding box is also added to the RoI for network training. Suppose there are N RoIs with GT bounding boxes added, and K GT bounding boxes. Calculate the IoU between RoI and GT, and divide RoI into positive and negative samples according to the IoU threshold of 0.5, where the positive sample label is 1 and the negative sample label is 0. Obtain the dictionary D corresponding to the i∈[0,N)th RoI and its most matching j∈[0,K)th GT bounding box related information. Finally, sample the training sample X for Mask Head with the ratio of positive samples to 1 / 4 of all samples.

[0183] Since Mask Loss is only defined on positive samples, it is also necessary to perform foreground screening on the training sample X. Assume that a total of M positive sample RoIs are obtained, and the i-th RoI i After the region of interest is aligned RoIAlign, a feature map F of size 14×14 is obtained i , F i After masking the header Mask Head, we get a 28×28 size prediction feature map pred iMask target gt_mask i The acquisition of GT is first to find the GT information (including category, border, polygon Mask) according to D through the RoI number, and then cut out the border area on the original image Mask according to the border information, and finally resize the cropped part to 28×28 to get the final gt_mask i The average binary cross entropy loss function is used to calculate L mask :

[0184]

[0185] in and is the feature map pred i and gt_mask i The eigenvalue at the (x,y) position;

[0186] Step 3-6, set the MaskIoU loss function: let the predicted value of the ith feature map after the cascade operation after the predicted mask intersection and comparison head MaskIoU Head be pred_maskiou i , the MaskIoU target is gt_maskiou i In steps 3-5, this value can be obtained based on the predicted Mask and the Mask information in the corresponding GT to obtain gt_maskiou i .use The loss function is used to calculate L maskiou :

[0187]

[0188] Step 3-7, set the multi-task loss function: add the above classification loss function, center loss function, distance regression loss function, Mask loss function and MaskIoU loss function to get the total multi-task loss function of the small batch:

[0189]

[0190] Where N cls 、N ctn 、N reg 、N mask 、N maskIoU is the normalization coefficient of the corresponding loss function, and N cls =N ctm =N reg is the number of positive samples in the predicted bounding box head, N mask =N maskIoUThe number of positive samples obtained after the proposed bounding box proposals are predicted in the prediction bounding box head according to the IoU threshold and the ratio of positive and negative samples. ctn , cls , cls , cls is the balance coefficient of each loss, and its value is 1;

[0191] Step 3-8, set the network learning related parameters and perform training: Use the pre-trained weights of VovNetV2-99 on ImageNet for transfer learning. Select the 1000 proposals with the highest confidence in the bounding box prediction results, and retain 100 proposals after non-maximum suppression, where the IoU threshold in non-maximum suppression NMS is 0.6. In Mask Head, proposals with an IoU value greater than 0.5 with the true marked box are regarded as positive samples, otherwise they are negative samples, and the number of positive samples used for training accounts for 1 / 4 of all training samples. At the same time, network training uses small batch stochastic gradient descent to optimize the network model, with a batch size of 2 and a learning rate of 0.0025. The Warmup strategy is used in the first 1000 iterations, and a momentum coefficient of 0.9 is used. In terms of regularization, 10 is used. -4 The weight decay coefficient. The entire training epoch is 24 rounds. In the 16th and 22nd rounds, the learning rate step reduction strategy is adopted with a reduction coefficient of 0.1. After setting the above learning parameters, use the enhanced training data in step 3-1 to train the constructed network;

[0192] Step 4: Use the trained instance segmentation network to predict the segmentation results, which specifically includes the following steps:

[0193] Step 4-1, perform data augmentation on the image that needs instance segmentation in the same way as in step 3-1, except that only a single scale is performed on the image during inference, that is, the long side is 1333 and the short side is 800;

[0194] Step 4-2, the trained network performs forward calculation on the test image obtained in step 1-3, removes proposals with confidence less than 0.05 in the predicted bounding box head, and screens out 1000 proposals with the highest confidence. A maximum of 50 RoIs are obtained through NMS for Mask branch prediction, and finally the kth Mask is obtained according to the category k predicted by the classification branch (but the data used in this article only contains one category). The 28×28 size Mask will be scaled to the corresponding RoI scale and binarized with a threshold of 0.5 to generate the final mask Mask;

[0195] In step 4-3, the top-50 proposals also use the MaskIoU header to predict the MaskIoU, which will be multiplied by the classification confidence to obtain the final Mask confidence.

[0196] Step 4-4, according to the Mask generated in step 4-2 and the Mask confidence obtained in step 4-3, NMS is used with an IoU threshold of 0.5 to filter out the final predicted Mask;

[0197] Step 4-5, segment the mesoscale convective system instance of the geostationary satellite infrared cloud images of the Jianghuai region of my country at adjacent times, and obtain the following Figure 5 Examples of mesoscale convective systems shown;

[0198] Step 5, based on the principle of relevant target matching, realizes the tracking of multiple continuous mesoscale convective systems, while taking into account the ubiquitous splitting and merging phenomena, which specifically includes the following steps:

[0199] Step 5-1, find the centroid coordinates of each mesoscale convective system instance Characteristic area, intensity P:

[0200]

[0201] area=N

[0202]

[0203] Where N is the total number of pixels contained in the mesoscale convective system instance, x i With y i are the x and y coordinates of the ith pixel, f(i) is the brightness temperature of the ith pixel, and f z is the accumulated sum of the lighting temperatures of each pixel in the example, and the image resolution (4km) of the embodiment of the present invention needs to be combined in subsequent calculations;

[0204] Step 5-2, calculate the duration, centroid position change, area change and intensity change of the mesoscale convective system at two adjacent moments according to the time sequence. If two mesoscale convective systems at adjacent moments satisfy the centroid position change ≤ 50m / s and area change ≤ 5km 2 / s, and the intensity change is ≤0.001℃ / s, it can be preliminarily judged as the same target;

[0205] Step 5-3, considering the common splitting and merging phenomena of mesoscale convective systems, corresponding processing is required. If there are multiple mesoscale convective systems MCS at a certain moment, they are denoted as X1, X2,…,X n and a certain MCS at the previous moment is denoted as P jIf the tracking matching principle in step 5-2 is met, then the phenomenon belongs to the splitting situation, and the X with the largest area after splitting is selected. m (1≤m≤n)ContinuationP j The trajectory of X m The index value of the previous moment of the action is j and the duration is P j The duration of the new cloud is +1, while other MCSs are regarded as the index value of the previous moment of the new cloud map is empty and the duration is initialized to 1;

[0206] If there are multiple MCSs at the previous moment, denoted as P, and a certain MCS at this moment is denoted as P a If the MCS tracking matching principle is met, the phenomenon belongs to the merging situation, and the P with the largest area at the previous moment is selected. m (1≤m≤z) as P a The last moment trajectory, update P m The index value of the previous moment of the action is m and the duration is P m The duration of the MCS that is not the largest in area at the previous moment ends its life cycle, and it is determined whether its duration is ≥ 1h. If so, the corresponding MCS behavior is obtained in reverse order according to the index value at the previous moment, thereby realizing MCS tracking.

[0207] Figure 6 In the tracking results shown in the figure, three cloud clusters are identified at the first moment with a half-hour interval between each moment. Similarly, three cloud clusters are identified at the second moment. The cloud cluster feature quantities (center of mass position movement distance, area change, and intensity change) at two adjacent moments are calculated and analyzed. The cloud clusters that correspond to the first moment and meet the target matching principle are marked at the second moment. Figure 6 At the second moment, it can be seen that the target cloud 3 identified at the first moment cannot find a cloud that meets the target matching principle at the second moment, so cloud 3 disappears. At the first moment, cloud 1 and cloud 2 have clouds that meet the target matching principle at the second moment, so the corresponding clouds at the second moment are marked with the same number as follows Figure 6 Cloud clusters 1 and 2 in the second moment. Cloud cluster 4 in the second moment is the newly formed cloud cluster at that moment. According to the clouds identified and tracked at the first moment and the second moment, the target cloud clusters tracked in the third and fourth periods can be obtained, which are Figure 6 The third and fourth moments are shown. Finally, according to the duration requirement of ≥1h, the MCS that meets the requirements can be screened. Figure 6 Only cloud cluster 2 is the correct MCS, which appears at the first, second, and third moments and lasts for a total of one hour (satisfying ≥1h);

[0208] This embodiment performs instance segmentation of mesoscale convective systems on the public geostationary satellite infrared brightness temperature data provided by the National Oceanic and Atmospheric Administration of the United States. The experimental configuration environment is shown in Table 1. It is compared with other current mainstream deep learning instance segmentation methods such as Mask R-CNN, Mask Soring R-CNN, Cascade Mask R-CNN, and HTC under the same configuration environment to obtain the experimental results shown in Table 2. The evaluation criteria adopted are part of the standards of the COCO dataset (i.e., boxAP, box AR, mask AP, mask AR, and inference time).

[0209] Table 1 Experimental configuration environment

[0210]

[0211] Table 2 Instance segmentation comparison experimental results

[0212]

[0213] It can be seen that the present invention has great advantages over some current mainstream instance segmentation methods in terms of accuracy and recall rate on this dataset, while maintaining a detection speed similar to that of Mask R-CNN without any additional overly complex computational overhead, which effectively proves the characteristics of the present invention with high detection accuracy and high speed.

[0214] The present invention provides a method for identifying and tracking a mesoscale convective system based on image anchor-free frame detection. There are many methods and approaches to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A method for identifying and tracking mesoscale convective systems based on image anchor-free frame detection, characterized in that: The following steps are involved: Step 1: preprocess the original infrared brightness temperature data file according to the relevant instructions of the satellite data: including cropping of the original satellite data, marking of image polygon instances, and random division of the training set, validation set, and test set; Step 2: Construct a convolutional neural network for instance segmentation of mesoscale convection systems based on anchor-free frames: it is divided into a backbone network for feature extraction, a feature pyramid network that integrates multi-scale features, a prediction frame network head for classification, distance frame regression, and prediction of centrality, a mask generation network head, and a network head for predicting MaskIoU; Step 3: Perform multi-scale image enhancement on the training set, and use transfer learning to supervise the training of the mesoscale convection system instance segmentation convolutional neural network built in step 2 to automatically learn network parameters: use small batch stochastic gradient descent method for training, and set the loss functions of the three network heads: predicting bounding boxes, generating masks, and predicting MaskIoU; Step 4, using the trained network to segment the mesoscale convective system instances on the geostationary satellite infrared cloud images at adjacent moments, and obtaining the relevant records of the mesoscale convective system at continuous moments; Step 5, based on step 4, tracking of the mesoscale convective system is achieved according to the relevant target matching principle; Step 2 includes: Step 2-1, build a backbone network for feature extraction: The backbone network in the instance segmentation network uses a 99-layer convolutional neural network VoVNetV2-99. The convolutional layers in the convolutional neural network VoVNetV2-99 are all in the form of Conv-BN-ReLU, that is, a combination of convolutional layer, batch normalization layer BN, and linear rectification function ReLU; at the same time, the convolutional neural network VoVNetV2-99 structural block uses cross-layer connection to achieve identity mapping. The residual block containing the identity mapping is defined as: Y=F(X,{W i })+X Where X and Y represent the input feature map and output feature map of each structural block respectively, F(X,{W i }) is the residual mapping function that needs to be learned; At the same time, the channel attention eSE mechanism is also introduced in the network residual block. The specific implementation method is as follows: A eSE (X div )=σ(W C (gap(X div ))) Where X div It is a diversified feature map obtained by cascading network feature maps and reducing their dimensionality. is the channel-level global average pooling, W and H are the width and height of the feature map X, respectively. i,j is the eigenvalue of feature graph X at (i, j), W C is the weight of the fully connected layer, σ represents the Sigmoid activation function, A eSE (X div ) is the calculated channel attention feature descriptor, which is consistent with X div Perform element-level multiplication to finally obtain the refined feature map X refine ; The specific steps to build the convolutional neural network VoVNetV2-99 include: Step 2-1-1, construct the network stem stage 1: This stage contains three convolutional layers: first, a convolutional layer with a convolution kernel size of 3×3, a stride of 2, a padding of 1, and an output channel number of 64 is used to downsample the input image, and then a convolutional layer with a convolution kernel size of 3×3 and an output channel number of 64 and a convolutional layer with a convolution kernel size of 3×3, a stride of 2, and an output channel number of 128 are connected; after the input image passes through this stage, the first scale feature map C1 is generated, and the feature map C1 has an Output Stride of 4 compared to the input image scale; Step 2-1-2, construct the network single aggregation module stage 2, this stage contains a residual block, the residual block contains 5 convolution layers with a convolution kernel size of 3×3 and an output channel of 128; after the input image is convolved, the 5 convolution layers each obtain a diversified feature map with a channel number of 128, and on the last convolution layer, the feature map of this layer is cascaded with the feature maps of the previous 4 layers and the input first scale feature map C1 to obtain a feature map with 128*6=768 channels, and then the convolution kernel size is 1×1, the step size is 1, the padding is 0, and the output channel number is 256. The convolution layer is used for dimensionality reduction to obtain a diversified feature map X div , combined with the channel attention eSE mechanism to obtain the final X refine , X refine It is also necessary to perform element-wise addition with the input feature map to achieve identity mapping. Finally, this stage obtains the second-scale feature map C2 with a channel number of 256. The scaling ratio of the feature map C2 to the input image scale is 4. Step 2-1-3, build the network single aggregation module stage 3: use a 3×3 maximum pooling layer with a step size of 2 and a padding of 0 to perform 2x downsampling, and use a 3×3 adjustable deformable convolution in the residual block. The adjustable deformable convolution is defined as: Where K is the total number of convolution kernel sampling locations, w k is the convolution kernel weight, p k is the predefined offset of the kth position relative to the center of the receptive field, x(p) and y(p) are the eigenvalues ​​of the input feature map x at position p and the output feature map y at position p, respectively, Δp k is the learnable offset of the kth position, Δm k ∈[0,1] is the adjustment number of the kth position; the adjustable deformable convolution is implemented by adding an additional convolution layer with the same spatial resolution and expansion rate as the conventional convolution to learn the offset and adjustment number of each position of the feature map in the x and y directions of the two-dimensional plane; The OSA module stage 3 contains 3 residual blocks, each of which contains 5 adjustable deformable convolution layers with a convolution kernel size of 3×3 and an output channel of 160. Like the network single aggregation module stage 2 in step 2-1-2, the feature map of this layer is cascaded with the feature maps of the first 4 layers and the input second-scale feature map C2 on the last convolution layer to obtain a diversified feature map with 256+160*5=1056 channels. This is the diversified feature map of the first residual block. The number of channels of the diversified feature map obtained after the cascade operation of the second and third residual blocks is 512+160*5=1312, and the number of output channels of the 1×1 conventional convolution layer with a step size of 1 and padding of 0 for dimensionality reduction after the feature map cascade is 512, which also includes the channel attention mechanism; after the OSA module stage 3 operation, the third-scale feature map C3 with a channel number of 512 is finally obtained, and the feature map C3 has an Output Stride of 8 compared to the input image scale; Step 2-1-4, construct the network single aggregation module stage 4: This stage contains 9 residual blocks, each residual block contains 5 adjustable deformable convolution layers with a convolution kernel size of 3×3 and an output channel of 192. Like the network single aggregation module stage 2 and the network single aggregation module stage 3, the feature map of this layer is cascaded with the feature map of the first 4 layers and the input third-scale feature map C3 on the last convolution layer to obtain a diversified feature map containing 512+192*5=1472 channels. , this is the diversified feature map of the first residual block. The number of channels of the diversified feature map obtained after the cascade operation of the second to the ninth residual blocks is 768+192*5=1728, and the number of output channels of the 1×1 conventional convolution layer with a step size of 1 and padding of 0 for dimensionality reduction after the feature map cascade is 768, which also includes the channel attention mechanism; after the network single aggregation module stage 4, the fourth scale feature map C4 is finally obtained. The scaling ratio of the feature map C4 to the input image scale is 16; Step 2-1-5, build the last structure of the backbone network, the network single aggregation module stage 5: this stage contains 3 residual blocks, each residual block contains 5 adjustable deformable convolution layers with a convolution kernel size of 3×3 and an output channel of 224. Like the network single aggregation module stage 2 and the network single aggregation module stage 3, the feature map of this layer is cascaded with the feature map of the previous 4 layers and the input fourth scale feature map C4 on the last convolution layer to obtain a multi-channel number of 768+224*5=1888. Sampling feature map, this is the diversified feature map of the first residual block. The number of channels of the diversified feature map obtained after the cascade operation of the second and third residual blocks is 1024+224*5=2144, and the number of output channels of the 1×1 conventional convolution layer with a step size of 1 and padding of 0 for dimensionality reduction after the feature map cascade is 1024, which also includes the channel attention mechanism; after this stage, the feature map C5 is finally obtained, and the scaling ratio OutputStride of the feature map C5 compared to the input image is 32; Step 2-2, combine the feature pyramid network FPN to fuse multi-scale features: The feature pyramid network FPN performs top-down and lateral connection on the different scale features {C3, C4, C5} obtained in step 2-1 to fuse the features to obtain {M3, M4, M5}, where M5 is obtained by passing the feature map C5 through a 1×1 convolution layer, M4 is obtained by performing element-wise addition of the feature map obtained by performing 1×1 convolution on the feature map C4 after upsampling the nearest neighbor by 2 times of M5, and M3 is obtained by performing element-wise addition of the feature map obtained by performing 1×1 convolution on the feature map C3 after upsampling the nearest neighbor by 2 times of M4; Finally, each layer in {M3, M4, M5} is convolved with a kernel size of 3×3 and an output channel number of 256 to obtain the feature layer {P3, P4, P5}; In the instance segmentation network, additional feature layers P6 and P7 are added. The P6 and P7 feature layers are obtained by downsampling P5 and P6 by 2 times through a 3×3 convolution layer with a stride of 2, and finally the feature layers {P3, P4, P5, P6, P7} are obtained; Step 2-3: Build the instance segmentation bounding box prediction head. This part contains two branches: a classification branch and a multi-task branch that performs center and distance bounding box regression in parallel. Specifically, it includes: Step 2-3-1, build the classification branch: after each feature layer in the feature pyramid network FPN, three regular convolutional layers with 256 input channels and 256 output channels and 3×3 convolution kernel size and an adjustable deformable convolutional layer are connected in sequence. These four convolutional layers no longer use batch normalization BN but group normalization GN; at the end of the classification branch, add a prediction classification convolutional layer with 256 input channels, 1 output channel and 3×3 convolution kernel size; Step 2-3-2, construct a multi-task branch for parallel regression of center and distance border: This part sequentially connects three regular convolutional layers with the same structure as the classification branch and an adjustable deformation convolutional layer after the feature pyramid network FPN feature layer of each scale. After these four convolutional layers, a convolutional layer containing two parallel branches is connected: a 3×3 convolutional layer with an output channel number of 4 for border regression, and the four dimensional values ​​output by this branch represent the distance from the current position to the four sides of the border; a 3×3 convolutional layer with an output channel number of 1 for predicting center, and the output of this branch is a dimension representing the center value; Assume that the distance between a sample point (x, y) on the feature map F and the four edges of the target border to which the point belongs is d = (l, t, r, b), where l, t, r, and b represent the distance from the sample point (x, y) to the left, top, right, and bottom of the rectangular border, respectively. Then the centrality is defined as: Among them, min() and max() are the minimum and maximum functions respectively; Step 2-4, constructing the instance segmentation network RoI head based on the anchor-free frame, specifically including: Step 2-4-1, construct the region of interest alignment RoIAlign layer: The RoIAlign layer first maps the RoI to the corresponding feature pyramid network FPN feature layer P according to the size of the region of interest RoI k The implementation is: k=Ceil(k max -log2(A input / A RoI )) Among them A input With A RoI Represent the area of ​​the input image and the area of ​​the RoI respectively; k max is the last layer of the backbone network, which is set to 5; Ceil() is the rounding function, and k is the number of feature pyramid network FPN feature layers mapped by RoI; The RoI Align layer divides the corresponding position area on the mapped feature map into 14×14=196 small areas of equal size, adaptively samples these areas, takes the center point position of each part, and then uses bilinear interpolation to calculate, finally obtaining a 14×14 fixed-size feature map; Step 2-4-2, construct a Mask head with a spatial attention mechanism: The Mask head contains four consecutive convolutional layers with a convolution kernel size of 3×3 and 256 input and output channels. The spatial attention module is introduced after the fourth convolutional layer. The implementation mechanism is: Where X i is the input feature map, A sag (X i )Spatial attention feature descriptor, P max With P avg They represent the maximum pooled feature map and the average pooled feature map in the channel dimension respectively. represents the cascade operation, F 3×3 is a 3×3 convolutional layer, σ is a Sigmoid function, is element-wise multiplication, X sag The feature map that is finally combined with spatial attention; Then X sag The deconvolution layer with a convolution kernel size of 2×2 and a stride of 2 is used for upsampling to obtain a feature map with a size of 28×28 and the same number of channels. The last convolution layer of the Mask head is a Mask prediction layer specific to each category. Since the detection target is the category of mesoscale convective system, the convolution kernel size of the Mask prediction layer is 1×1, the stride is 1, and the number of output channels is 1. Step 2-4-3, construct the MaskIoU header and use the mask score Mask Scoring to re-represent the Mask quality; Step 2-4-3 includes: based on step 2-4-2, the output feature map of the Mask prediction layer is downsampled by 2 times using a 2×2 maximum pooling layer, and then cascaded with the RoI feature map of size 14×14 and number of channels 256 output by the RoI Align layer to obtain a feature map of size 28×28 and number of channels 257; The MaskIoU head contains four consecutive convolutional layers with a kernel size of 3×3 and an output channel number of 256. The stride of the last convolutional layer is 2, followed by two fully connected layers with an output channel number of 1024 and one with an output channel number of 1.

2. The method according to claim 1, characterized in that Step 1 includes: Step 1-1, read each original geostationary satellite infrared brightness temperature data file according to the relevant instructions of the satellite data, the file is an array in the format of 2×3298×9896, and obtain two grayscale cloud images at the corresponding time 0 and 30 minutes; Step 1-2, cropping the grayscale cloud image obtained in step 1-1: cropping each grayscale cloud image into two or more grayscale infrared cloud sub-images with a size of 420×360; Steps 1-3: give polygon instance-level annotations of mesoscale convective systems for each grayscale infrared cloud subimage, filter out subimages without mesoscale convective systems, and obtain a json file corresponding to each grayscale infrared cloud subimage. The entire annotation has only one category; and randomly divide the grayscale infrared cloud subimages into training set images, validation set images, and test set images in a ratio of 6:2:2, thereby forming a training set, a validation set, and a test set.

3. The method according to claim 2, characterized in that Step 3 includes the following steps: Step 3-1, convert the grayscale images in the training set into RGB three-channel images. The values ​​of the three RGB channels are the same, which are all grayscale values ​​of the grayscale images. Perform data enhancement on the converted images in the training set and the corresponding instance annotations: first, scale the images according to multiple scales, where the long side is 1333 and the short side is a random one of {640, 672, 704, 736, 768, 800}, and keep the original image ratio. At the same time, random horizontal flipping is also used for data enhancement; The pixel values ​​are centered and the image is normalized according to the mean in the ImageNet dataset. The normalization operation is performed on a channel basis. The mean values ​​of the R, G, and B channels are 123.675, 116.28, and 103.53, respectively, and the standard deviations are 58.395, 57.12, and 57.375, respectively. The scalar combination of the mean values ​​of the three RGB channels is represented as a vector The standard deviation scalar combination of the three RGB channels is represented as a vector σ. Let the input image be x, then the normalized image data x′ is Then fill the image scale to a multiple of 32; Accordingly, after the input image is scaled and horizontally flipped for data enhancement, the corresponding instance annotations must also undergo the same transformation to obtain the correct annotations; Step 3-2, set the classification loss function for predicting the border head in the instance segmentation network: Let the feature map F i (i=3,4,...7) The position (x i ,y i )The predicted value is set up in Indicates whether it is a positive or negative sample, Indicates the position (x i ,y i ) is a positive sample, Then it means the position (x i ,y i ) is a negative sample, is the position (x i ,y i ) is the probability of a positive sample; The position (x i ,y i ) is mapped to the corresponding position (x', y') of the input image as follows: where s i is the feature map F i Compared to the input image scale ratio Output Stride, if (x', y') falls into any real label border, it is a positive sample with Y = 1, otherwise Y = 0; Focus loss function based on α balance It is expressed as: Where α is the weight factor, and γ ≥ 0 is an adjustable focusing parameter; For the feature map F i , its total classification loss function for: Step 3-3, set the centrality loss function: given the feature map F i The positive sample feature point (x i ,y i ) distance bounding box regression target in They represent the feature points (x i ,y i ) to the left, top, right, and bottom of the real border; then the feature point (x i ,y i )’s centrality target Defined as: Let the feature map F i After the centrality branch, we get the position (x i ,y i ) Then the feature map F i The total center loss function for: in Indicates whether it is a positive or negative sample, Indicates the position (x i ,y i ) is a positive sample, Then it means the position (x i ,y i ) is a negative sample; For indicator functions, if the condition in parentheses If it holds, the indicator function value is 1, otherwise its value is 0; Step 3-4, set the distance regression loss function: The regression task uses the GIoU loss function, set the distance in They represent the feature points (x i ,y i ) to the left, top, right, and bottom of the predicted bounding box; the regression distance target is in They represent the feature points (x i ,y i ) to the left, top, right, and bottom of the real border; and Considered as an external bounding box, the GIoU loss function is expressed as: Where C is and The bounding box of the two, | | represents the area of ​​the region, Indicates that area C does not contain and Some areas, for and The intersection ratio of for and The generalized intersection and union ratio of ; Feature map F i Total distance regression loss function for: in is the element-level weighting coefficient of the loss function, which is the value of the position (x i ,y i )Centerness regression target; Step 3-5, set the Mask loss function: suppose the predicted bounding box head has a total of M RoIs, the i-th RoI i After aligning the region of interest RoIAlign and masking the header Mask Head, the predicted feature map pred is obtained i , and get RoI based on the classification and regression branches i Category and Mask target gt_mask i , using the average binary cross entropy loss function to calculate L mask : in and is the feature map pred i and gt_mask i The eigenvalue at the (x,y) position; Step 3-6, set the MaskIoU loss function: let the predicted value of the ith feature map after the cascade operation after the predicted mask intersection and comparison head MaskIoU Head be pred_maskiou i , the MaskIoU target is gt_maskiou i , using l2 loss function to calculate L maskiou : Step 3-7, set the multi-task loss function; Step 3-8: Set network learning related parameters and perform training.

4. The method according to claim 3, characterized in that Step 3-7 includes: adding the classification loss function, the center loss function, the distance regression loss function, the Mask loss function and the MaskIoU loss function to obtain a small batch total multi-task loss function L: Where N cls 、N ctn 、N reg 、N mask 、N maskIoU is the normalization coefficient of the corresponding loss function, and N cls =N ctm =N reg is the number of positive samples in the predicted bounding box head, N mask =N maskIoU is the number of positive samples obtained according to the IoU threshold and the ratio of positive and negative samples after the proposed bounding box proposals are predicted in the predicted bounding box head; ctn , cls , cls , cls is the balance coefficient of each loss.

5. The method according to claim 4, characterized in that Steps 3-8 include: using the pre-trained weights of VovNetV2-99 on ImageNet for transfer learning, selecting the 1000 proposals with the highest confidence in the pre-frame detection results, retaining 100 proposals after non-maximum suppression, where the IoU threshold in non-maximum suppression is 0.6; in Mask Head, proposals with an IoU value greater than 0.5 with the true marked box are regarded as positive samples, otherwise they are negative samples, and the number of positive samples used for training accounts for 1 / 4 of all training samples; at the same time, the network training uses small batch stochastic gradient descent to optimize the network model, with a batch size of 2 and a learning rate of 0.0025, and uses the Warmup strategy in the first 1000 iterations, and uses a momentum of 0.9; in terms of regularization, 10 -4 The weight decay coefficient of the training epoch is 24 rounds. In the 16th and 22nd rounds, the learning rate step reduction strategy is adopted with a reduction coefficient of 0.

1. After setting the above learning parameters, the constructed network is trained using the training set processed in step 3-1.

6. The method according to claim 5, characterized in that Step 4 includes the following steps: Step 4-1, the image that needs instance segmentation is processed in the same way as in step 3-1, except that the image is only scaled to a single scale during inference, that is, the long side is 1333 and the short side is 800; Step 4-2: The trained network performs forward calculation on the test image obtained in step 1-3, removes the proposed bounding box proposals with a confidence level less than 0.05 in the predicted bounding box head, and selects 1,000 proposed bounding box proposals with the highest confidence level. 50 of them are used for Mask branch prediction through non-maximum suppression, and finally the kth Mask is obtained according to the category k predicted by the classification branch. The 28×28 size Mask will be scaled to the corresponding proposal scale and binarized to generate the final mask Mask. Step 4-3: The top-50 proposals also use the MaskIoU header to predict the MaskIoU, which will be multiplied by the classification confidence to get the Mask confidence; Step 4-4, according to the Mask generated in step 4-2 and the Mask confidence obtained in step 4-3, NMS is used with an IoU threshold of 0.5 to filter out the final predicted Mask; Step 4-5, segment the mesoscale convective system instances on the geostationary satellite infrared cloud images of the Jianghuai region of my country at adjacent moments, and obtain the mesoscale convective system instances at continuous moments.

7. The method according to claim 6, characterized in that Step 5 includes the following steps: Step 5-1, find the centroid coordinates of each mesoscale convective system instance Characteristic area, intensity P: area=N Where N is the total number of pixels contained in the mesoscale convective system instance, x i With y i is the horizontal and vertical coordinates of the i-th pixel, f(i) is the brightness temperature of the i-th pixel, f z It is the accumulated sum of the lighting temperatures of each pixel in the instance; Step 5-2, calculate the duration, centroid position change, area change and intensity change of the mesoscale convective system at two adjacent moments according to the time sequence; if the two mesoscale convective systems at adjacent moments meet the centroid position change ≤ 50m / s and area change ≤ 5km 2 / s, and the intensity change is ≤0.001℃ / s, it is preliminarily judged to be the same target; Step 5-3: If there are n mesoscale convective systems MCS at a time, denoted by X1, X2,…,X n The MCS at the previous moment is denoted as X j If the tracking matching principle in step 5-2 is met, then the phenomenon belongs to the splitting situation, and the X with the largest area after splitting is selected. m Continuation X j The trajectory of X, 1≤m≤n; update X m The index value of the previous moment of the action is j and the duration is updated to: X j The duration of the new cloud is +1, while other MCSs are regarded as the index value of the previous moment of the new cloud map is empty and the duration is initialized to 1; If there are more than two MCSs at the previous moment, denoted as X1, X2,…,X z And an MCS at that time is denoted as X a If the MCS tracking matching principle is met, the phenomenon belongs to the merging situation, and the X with the largest area at the previous moment is selected. m As X a The last moment trajectory, 1≤m≤z, update X m The previous index value of the action is m and the duration is updated to: X m The duration of the MCS with the non-largest area at the previous moment ends, and its duration is determined to be ≥ 1h. If so, the corresponding MCS behavior is obtained in reverse order according to the index value at the previous moment, thereby realizing MCS tracking.

Citation Information

Patent Citations

  • Method for integrating models of a vehicle health management system

    CN103093077A

  • Thunderstorm cloud cluster identification method and system for meteorological satellite cloud picture

    CN106023177A

  • A mesoscale convection dynamic tracking method based on new generation of stationary satellite data

    CN109816684A

  • Systems and methods for outage prediction

    WO2018013148A1