Method and system for segmenting CME and automatically tracking and three-dimensionally reconstructing CME based on deep learning

Abstract

The present invention discloses a method and system for segmenting CMEs based on deep learning and automatically tracking and reconstructing CMEs in three dimensions, which are mainly divided into four modules: classification, segmentation, tracking and 3D reconstruction. A pre-trained classification model is used to classify the coronal images and divide the preliminary CME image sequence. Then, the observation data is collected to mark the structural areas of the CME to construct a CME semantic segmentation dataset. The Transformer-based segmentation model is trained to achieve the best results in the semantic segmentation of CME images. In addition, the tracking algorithm is improved to solve the task of separating multiple CMEs. In the last module, the tracking results are combined with the polarization ratio technology to develop the first single-view three-dimensional CME catalog without the need for manual mask annotation. The present invention provides higher accuracy in the 2D CME automatic catalog and provides more reliable CME physical parameters, including 3D propagation direction and speed, which can be applied to any coronal image data with polarization measurement capabilities.

Description

Technical Field

[0001] The present invention belongs to the field of computer image processing and computer vision, and in particular to a method and system for segmenting CME based on deep learning and automatically tracking and three-dimensionally reconstructing CME. Background Art

[0002] Coronal mass ejections (CMEs) are one of the most violent and largest solar activity phenomena in the solar atmosphere, releasing a large amount of magnetic field plasma in a short period of time. CME observed in a white-light coronagraph is a new, discontinuous, bright, white-light feature with a radially outward velocity. When the CME direction is toward the Earth, it may cause geomagnetic storms and bring about space weather disasters. The impact of space weather can be felt in many aspects of civilian life, commerce, and national security, including communications, navigation, power grids, and satellite operations. Therefore, the tracking and three-dimensional reconstruction of CMEs are not only important for physical research, but also for monitoring the space weather environment.

[0003] Thanks to decades of observations from the Solar and Heliospheric Observatory (SOHO), especially the Large Angle Spectroscopic Coronagraph (LASCO), we have been able to study CMEs and their relationship with the solar cycle. LASCO contains three white-light coronagraphs with different fields of view. Among them, the field of view of LASCO C2 is about 2-6R. In order to further understand CMEs, especially their three-dimensional evolution, the Solar Terrestrial Relations Observatory (STEREO) spacecraft is equipped with the Solar Terrestrial Relations Coronal and Heliospheric Observatory (SECCHI). However, with the continuous accumulation of coronal images, it has become particularly important to identify and catalog CMEs. The Coordinated Data Analysis Workshop (CDAW) is a widely used catalog, but it relies on observers to manually record CME events, which is subjective and time-consuming, especially during the period of solar maximum activity. The detection and tracking of CMEs requires a lot of manpower. The above shortcomings of manual CME catalogs have prompted the development of automatic catalogs.

[0004] Studying the three-dimensional structure of CMEs helps us understand the possible relationship between the initial evolution of CMEs, the subsequent propagation of ICMEs, and their impacts on the Earth. Some scholars have developed polarization ratio techniques to infer the position distribution of CME structures in three-dimensional space. Polarization ratio technology is used for single-view polarization Thomson scattering images to locate the center of mass along the line of sight (LOS) and estimate its density distribution. The advantage of this method is that it only relies on observations from one perspective, but currently these methods using polarization ratios require manual extraction of the CME structure area, and there is no automated method to obtain a catalog of CME three-dimensional reconstructions. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a method and system for segmenting CME based on deep learning and automatically tracking and three-dimensionally reconstructing CME.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] The method for segmenting CME and automatically tracking and three-dimensionally reconstructing CME based on deep learning is characterized by comprising the following steps:

[0008] Step 1: Collect the observation data of the white-light coronagraph, pre-process the observation data, and obtain the corona differential image;

[0009] Step 2: Use the pre-trained convolutional neural network model to perform binary classification on the coronal difference images, infer whether a single coronal difference image contains CME, and divide the preliminary CME events according to the spatiotemporal continuity rule;

[0010] Step 3: Train the image segmentation model to perform pixel-by-pixel binary classification on the coronal difference images in CME events, infer whether each pixel has a CME structure, and predict the CME structure area of ​​a single coronal difference image;

[0011] Step 4: Based on the prediction results of the CME structure area, define tracking rules to track the CME structure evolution;

[0012] Step 5: Based on the polarization ratio method, calculate the position distribution of CME in three-dimensional space according to the tracking results.

[0013] To optimize the above technical solutions, the specific measures taken also include:

[0014] Furthermore, in step 1, the coronal difference image is obtained by subtracting the previous frame from the current frame.

[0015] Furthermore, in step 2, the VGG convolutional neural network model is used to perform binary classification on the input coronal difference image as to whether or not CME exists, and a predicted label of the coronal difference image is output, where a label of 1 indicates the presence of CME and a label of 0 indicates the absence of CME; based on the output predicted label, the input coronal difference image is divided into multiple CME image sequences in accordance with the spatiotemporal continuity rule to represent preliminary CME events.

[0016] Furthermore, before dividing the input coronal difference image into multiple CME image sequences, the CME images containing two consecutive coronal difference images with labels of 0 are deleted; the time difference between each CME image sequence is calculated, and a time threshold is set for comparison, and according to the comparison result, it is decided whether to merge the current CME image sequence into the previous CME image sequence or the next CME image sequence or keep it independent.

[0017] Furthermore, in step 3, the image segmentation model adopts the Transformer-based semantic segmentation network model SegFormer, which uses the coronal difference image in the CME event as input and outputs the CME structure area; the encoder of SegFormer consists of four different transformer blocks, which output four feature maps X of different dimensions. i ,i∈1,2,3,4; the processing formula of SegFormer's decoder is as follows:

[0018]

[0019] X=MLP (4C,C) (Concat([X''1,X''2,X''3,X''4]))

[0020]

[0021] M=Upsample H×W (M p )

[0022] In the formula, MLP stands for multi-layer perceptron, the subscript represents dimension conversion, and C i represents the dimension of the i-th MLP, C represents the dimension unified by the linear layer, Upsample represents upsampling, Concat represents feature fusion, H and W represent the height and width of the input image respectively, and N class represents the number of categories, and M represents the CME structure area.

[0023] Furthermore, the step 4 specifically includes the following sub-steps:

[0024] Step 4-1: Perform image segmentation and post-processing on the image of the CME structure area;

[0025] Step 4-2: Convert the processed image to the polar coordinate system, count the angular width, fit the velocity, select the CME structure area within the angular width and the image that meets the velocity rule, and obtain the CME image sequence as the tracking result.

[0026] Furthermore, in step 4-1, the image segmentation post-processing specifically includes: removing invalid and erroneous images, distinguishing image sequences based on temporal continuity, and distinguishing image sequences based on spatiotemporal continuity;

[0027] The method of removing invalid and erroneous images specifically comprises: deleting the noise image, performing polar coordinate transformation on the remaining images, starting from the due north direction of the sun and sequentially unfolding in a counterclockwise direction, transforming each image at a resolution of θ×r, where θ is the polar angle and r is the radial distance measured from the center of the sun, counting the CME structure at each position angle, and discarding the image if the number of position angles containing the CME structure is less than a set value;

[0028] The method of distinguishing image sequences based on time continuity specifically includes: for an image sequence exceeding a set number of frames, calculating the time difference between two adjacent images, and dividing the image sequence by comparing it with the set time value;

[0029] The method of distinguishing image sequences based on spatiotemporal continuity is specifically as follows: converting an image sequence exceeding a set number of frames into a polar coordinate system, identifying the propagation angle of the CME structure by comparing the height difference between consecutive images, judging whether the CME structure propagates radially outward according to the propagation angle, marking the CME structure that propagates radially outward, and on this basis, using a marking morphological operator to process it into connected areas with different color marks, the boundaries of the same color areas are marked by bounding boxes, merging overlapping bounding boxes, and dividing the image sequence according to the bounding boxes.

[0030] Furthermore, in step 4-2, the statistical angular width is specifically: calculating the maximum height at each position angle, and merging the discrete position angles into one or more angular widths accordingly; the fitting speed includes the median speed, the maximum speed and the acceleration.

[0031] Furthermore, the step 5 specifically includes the following sub-steps:

[0032] Step 5-1: Obtain polarization brightness and total brightness;

[0033] Step 5-2: According to the tracking results, the CME area is obtained using the convex hull method, and then the edge of the CME area is smoothed using a circular filter to obtain the mask of the CME area;

[0034] Step 5-3: Calculate the position of the CME in three-dimensional space by calculating the ratio of the polarized brightness to the total brightness in the mask.

[0035] Accordingly, the present invention also proposes a system for segmenting CME based on deep learning and automatically tracking and three-dimensionally reconstructing CME, which is characterized by comprising:

[0036] The image preprocessing module is used to collect the observation data of the white-light coronagraph, preprocess the observation data, and obtain the corona differential image;

[0037] The image classification module is used to pre-train the VGG convolutional neural network model, perform binary classification on the coronal difference images, infer whether a single coronal difference image contains CME, and divide the preliminary CME events according to the rules of spatiotemporal continuity;

[0038] The image segmentation module is used to train the image segmentation model, perform pixel-by-pixel binary classification on the coronal difference image in the CME event, infer whether each pixel has a CME structure, and predict the CME structure area of ​​a single coronal difference image;

[0039] The CME tracking module defines tracking rules to track the CME structure evolution based on the prediction results of the CME structure area;

[0040] The 3D reconstruction module calculates the position distribution of CME in 3D space according to the tracking results based on the polarization ratio method.

[0041] The beneficial effects of the present invention are as follows: the present invention adopts the Transformer-based image segmentation model to segment CME for the first time. The SegFormer model can extract fine-grained CME features and show better robustness and generalization in COR1-A-based image visualization. The new tracking algorithm adopted by the present invention successfully solves the difficulty of distinguishing multiple CMEs from image sequences, reduces the missed detection and false detection rates, and provides more reliable CME physical parameters. The method of automatic single-perspective reconstruction of three-dimensional CME proposed in the present invention no longer relies on manual annotation of CME areas. Using the method and system proposed in the present invention, an automated catalog of two-dimensional CMEs can be developed; for white-light coronagraphs equipped with polarization instruments, an automated catalog of three-dimensional CMEs can also be developed. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Flow chart of the method for segmenting CME based on deep learning and automatically tracking and three-dimensionally reconstructing CME.

[0043] Figure 2 Schematic diagram of partial annotation results of the CME semantic segmentation dataset.

[0044] Figure 3 This is the architecture diagram of the Transformer-based image segmentation model SegFormer.

[0045] Figures 4a-4f It is a process diagram of the tracking phase, where: Figure 4a Indicates the red segmented CME area, Figure 4b represents the gray area after conversion, Figure 4c represents the fitting speed, Figure 4d represents the propagation angle of the CME structure identified by comparing the height difference between consecutive images, Figure 4e Indicates the use of morphological closing operations to reduce discrete points. Figure 4f Schematic diagram showing the results of distinguishing multiple CMEs in an image sequence.

[0046] Figure 5 Provide multi-view visualization of CME in three-dimensional space; DETAILED DESCRIPTION

[0047] The present invention will now be described in further detail with reference to the accompanying drawings.

[0048] In one embodiment, if Figure 1 As shown, the present invention proposes a method for segmenting CME based on deep learning and automatically tracking and three-dimensionally reconstructing CME, which specifically includes the following steps.

[0049] Step 1: Image preprocessing stage: Collect the original white-light coronagraph observation data, preprocess the observation data, and obtain the corona differential image.

[0050] Step 1-1: Use lasco_readfits.pro and reduce_level.pro in the Solar Physics Software Package (SSW) to process the 0.5-level LASCO C2 Fits file into level 1 data. The processing includes dark current, flat field, stray light, distortion, vignetting, photometry, time and position. The image size is then downsampled from 1024×1024 to 512×512. Next, a 3×3 mean filter is used for these downsampled images to suppress sharp noise. This filter works by replacing the value of each pixel with the average of the values ​​of its surrounding neighbors. Finally, the coronal difference image is obtained by subtracting the previous frame from the current frame.

[0051] Step 1-2: Similar to the LASCO C2 preprocessing steps to obtain the difference image, the COR1-A image was calibrated using secchi_prep.pro in the Solar Physics Software Package (SSW) to process the data from magnitude 0.5 to magnitude 1. Then, a mean filter was used to suppress the noise.

[0052] Step 2: Image classification stage.

[0053] Step 2-1: Use the pre-trained VGG model to classify the input difference image into two categories: the presence or absence of CME. The model outputs a label of 0 or 1: 1 indicates the presence of CME in the image, while 0 indicates the absence of CME. Considering the similarity between the difference images of LASCO C2 and SECCHI COR1-A, the classification model trained on LASCO C2 is generalized to COR1-A. The predicted label of the COR1-A corona difference image is obtained.

[0054] Step 2-2: CME events are temporal and their structures evolve over time. Based on the predicted labels obtained, rules are formulated to divide these images into CME image sequences, which represent preliminary CME events. Note that in this step, the CME image sequence can contain one or more separate CME events. Each image sequence will be divided into separate CMEs in the tracking module. Next, the spatiotemporal continuity rule is followed to obtain preliminary CME events. The rule contains the following points: the CME image sequence can contain images labeled 0 (no CME), but cannot contain more than two consecutive images labeled 0. An image labeled 0 may be a misjudgment of the classification model. However, the probability of two consecutive misclassifications is low. Define the lower limit of the total time and number of images in the sequence to determine whether to discard the image sequence. Finally, calculate the time difference between the image sequences, and set a time threshold of one hour to decide whether to merge the retained image sequence into the previous image sequence or the next image sequence, or keep the image sequence independent.

[0055] Step 3: Image segmentation stage.

[0056] Step 3-1: Construct a CME image segmentation dataset. CMEs can be divided into four categories according to their angular width: Narrow (width <= 30°), Limb (30° < width <= 120°), Partial Halo (120° < width < 360°), and Full Halo (width = 360°). A total of 375 LASCO C2 images were collected, covering CMEs of these four angular width types. Figure 2 As shown in the figure, the CME area and shock wave were manually marked in the difference image, and a CME image segmentation dataset was constructed.

[0057] Step 3-2: The image segmentation model SegFormer is trained on the CME semantic segmentation dataset based on LASCO C2 observation data. SegFormer is an encoder-decoder semantic segmentation model based on the transformer architecture. Figure 3 As shown in Figure 2, the improved SegFormer segmentation model uses the difference image as input and predicts the structure of CME as output. The structure of SegFormer can be divided into two components: encoder and decoder.

[0058] The encoder of the hybrid transformer consists of four different transformer blocks. There are three modules in each transformer block: Overlap Patch Merging, Mix feed-forward network (Mix-FFN), and Efficient Self-Attention (ESA). Overlap Patch Merging uses convolutional layers to reduce the resolution of feature maps. Interpolation is required to encode position information of different resolutions. Therefore, Mix-FFN replaces the position encoding in ViT with a 3×3 convolutional layer. The accuracy loss is avoided by implicitly learning the patch position information. ESA is an improvement based on traditional self-attention, reducing the computational complexity from O(N 2 ) is reduced to This is particularly beneficial for processing large resolution images. Traditional self-attention takes a query Q, a keyword K, and a value V as input. The dimensions of Q, K, and V are d k They are all N×C, where N=H×W is the sequence length, H is the height of the input image, and W is its width. The formula is as follows, where the softmax function is used to obtain the weights of the values.

[0059]

[0060] Compared with the traditional CNN encoder, SegFormer's multi-level encoder has a larger effective receptive field. SegFormer adopts a lightweight decoder consisting of only a multi-layer perceptron layer and reduces the amount of computation. First, each feature map X output by the encoder i (i∈1,2,3,4) are unified to the same dimension C through a linear layer. The second step is to upsample all these feature maps to Next, a linear layer is used to concatenate the connected features together. In the fourth step, another linear layer is used to predict the fused features. The resolution mask (mask), where N class Represents the number of categories. The formula of SegFormer decoder is as follows:

[0061]

[0062] X=MLP (4C,C) (Concat([X''1,X''2,X''3,X''4]))

[0063]

[0064] M=UpsampleH×W (M p )

[0065] In the formula, MLP stands for multi-layer perceptron, the subscript represents dimension conversion, and C i represents the dimension of the i-th MLP; Upsample represents upsampling, Concat represents feature fusion, and M represents the CME structure area.

[0066] In this example, the output layer is modified to change the last output layer of the SegFormer architecture to 512×512×2, corresponding to the height H, width W and number of categories N, respectively. class The output unit is normalized to obtain the probability of each pixel having a CME structure or not

[0067]

[0068] in, and is the output unit of each pixel in the output layer. The larger the value, the higher the probability that the pixel has a CME structure. The larger the value, the higher the probability that the pixel has no CME structure. Pixels with values ​​greater than 0.5 are considered to be pixels with CME structures detected. Unlike image classification, which only classifies the entire image, semantic segmentation classifies each pixel.

[0069] In the CME image segmentation training phase, the Adam optimizer with adaptive learning rate was used to train the model. Data augmentation was performed on the images of the LASCO C2 segmentation dataset to enhance the robustness and generalization of the model, such as random rotation, flipping, adding noise, Gaussian blur, changing brightness and contrast. The loss function uses cross entropy loss, and the formula is as follows:

[0070]

[0071] Where L is the loss value, N is the number of training images, and y i represents the true label value of image i, a i is the model predicted label value of image i calculated by the softmax function.

[0072] In the CME image segmentation test phase, similar to the CME image classification phase, the SegFormer model pre-trained on the LASCO C2 dataset is applied to the COR1-A data. The CME image sequence of COR1-A is input into the model, and the CME region can be segmented without fine-tuning. Most images without CME structures are removed during the segmentation phase. The purpose of the segmentation module is to extract the regions containing CME structures as accurately as possible and remove those images with incorrect classification predictions.

[0073] Step 4: CME tracking phase: Define tracking rules to track the structural evolution of CME and calculate relevant physical parameters such as the velocity, angular width, and central position angle of CME events.

[0074] Step 4-1: Image segmentation post-processing Before starting tracking, it is necessary to do a post-processing on the results obtained by the segmentation module to improve the tracking accuracy.

[0075] Image segmentation post-processing includes the following steps:

[0076] Step 4-1-1: Remove invalid or erroneous images. Each binary image of the CME boundary is obtained using a convex hull morphological operation. Binary images with less than 100 white pixels are deleted, which are considered to be dominated by noise. Next, the non-deleted images are polar transformed, starting from the north direction of the sun and unfolding sequentially in a counterclockwise direction. Each segmented image is transformed with a resolution of θ×r to count the CME structure at each position angle (PA), where θ is the polar angle and r is the radial distance measured from the center of the sun. Similarly, if the number of position angles containing CME structures is less than 5, the segmented image is discarded.

[0077] Step 4-1-2: Distinguish image sequences based on temporal continuity. In the COR1 FOV, a CME with a velocity of 250 km / s can be considered as a relatively slow CME. This velocity corresponds to a duration of two hours in the COR1 FOV, which covers 24 frames considering that the COR1 temporal resolution is 5 minutes. Therefore, a CME image sequence with more than 24 frames can be considered to contain multiple CME events. For such an image sequence, the time difference T between two adjacent images is calculated. diff If T diff >2h, the image sequence is divided into multiple sub-image sequences. This step is used as a rough separation of the CME sequence.

[0078] Step 4-1-3: Distinguish image sequences based on spatiotemporal continuity. The classification module and simple tracking algorithm have difficulty distinguishing multiple CMEs in an image sequence, especially when the position angles overlap but the CMEs erupt at different times. CMEs that erupt later are often missed, especially in the year of solar maximum activity. To solve this problem, this embodiment proposes a method for distinguishing sub-image sequences by combining the temporal and structural features of CMEs. Image sequences with more than 24 frames are converted to the [θ, r] polar coordinate system. Figures 4a-4f Provides an example that describes Figure 4a The red color divides the CME area into Figure 4b The transformed image has a resolution of 360×360 and a 1.3-4R ⊙ The radial field of view of the CME image sequence is i In the i ,θ j ] indicates that in PAθ j The maximum height of the CME mask at . Figure 4d The figure shows how to identify the propagation angle of CME structures by comparing the height difference between consecutive images. The white pixels in each row indicate that the CME structure at that position will propagate radially outward at the next moment. In order to reduce the discrete points, Figure 4e The morphological closing operation is used in. If propagating outward, it is marked. Subsequently, the marking morphological operator is used to identify connected regions with different color labels and remove small areas. The boundaries of the same color regions are identified by bounding boxes. Highly overlapping bounding boxes are merged. The time corresponding to the first image of the bounding box is used as the starting time T of the new sub-image sequence start The image corresponding to the maximum angle represents the moment when the CME angular width reaches its peak, denoted as T peak .like Figure 4f As shown, the improved tracking algorithm can successfully solve the challenging problem of distinguishing multiple CMEs in an image sequence.

[0079] Step 4-2: Track the CME phase.

[0080] Step 4-2-1: For easy tracking, transform the image into polar coordinate system.

[0081] Step 4-2-2: Count the angular widths. Calculate the maximum height at each PA, and then combine these discrete PAs into one or more angular widths. It is defined as the number of frames in the image sequence where the maximum height of each PA exceeds the minimum height threshold. The rule previously used to determine the minimum height threshold was based on half the sum of the maximum and minimum fields of view. However, this resulted in many CME events being missed in the COR1 images. After careful adjustment, the minimum height threshold was set to 2.3R⊙ , R ⊙ represents the solar radius. To identify the presence of a CME at a specific PA, if (L θ is a 1×360 vector, PAθ j Where ) is at least 2, assign the value 1 to Otherwise, it is marked as 0.

[0082] Step 4-2-3: Fitting velocity. Due to various factors, such as the evolution of CME structure, noise, and segmentation error, the change in radial height along the PA is sometimes non-increasing. Figure 4b As shown by the red line in , the height curve is fitted to the PA–height graph. In addition, the velocity calculation rule is modified so that the radial height in the time series does not need to increase continuously, but some radial heights are allowed to decrease slightly. In this way, a linear fit is performed on the points that meet the rule to more accurately calculate the velocity, as shown in Figure 4c As shown. By calculating the velocity distribution of the PA around the central PA, the median and maximum values ​​are taken as the median velocity and maximum velocity of the CME, respectively. The end time of the CME is difficult to determine, and the moment when the CME reaches the maximum height or runs out of the COR1 FOV is taken as the end time.

[0083] Step 4-2-4: CME image sequence of the tracking result. Select images of the CME structure area that is only within the angular width from the segmented mask, and select those images corresponding to the points that meet the fitting speed rule. In this way, a CME image sequence that meets the conditions can be obtained.

[0084] In short, this step provides the following basic parameters for the tracked CME: first appearance time and end time in COR1 FOV, center PA, angular width, median velocity, maximum velocity, and acceleration. The code has been rewritten in a more efficient and robust way. The improved tracking algorithm provides more reliable physical CME parameters.

[0085] Step 5: 3D reconstruction stage. Figure 5 As shown in Figure 1, based on the polarization ratio method, the position distribution of CME in three-dimensional space is calculated based on the tracking results. Unlike other methods such as forward modeling, the advantage of polarization ratio is that it only relies on observations from a single perspective to reconstruct the three-dimensional structure of CME.

[0086] Step 5-1: Obtain polarized brightness and total brightness. This technique is based on the well-known Thomson scattering theory, according to which a relationship can be established between the polarization degree of white light scattered by coronal electrons and the electron position, for example, along a given sky plane, the lower the polarization degree of the scattered light, the farther the electron is from the sky plane. The polarization degree of coronal white light can be calculated by coronal polarimetry, and then the weighted average distance of coronal electrons along the sky plane is estimated. Coronal polarimetry is a collection of three images observed at three different polarizer directions. From the three polarized coronagraph images, the polarized brightness and total brightness (pB and tB) can be calculated, and thus the polarization degree (the ratio between pB and tB) can be calculated. It should be noted that before deriving pB and tB from the polarized coronagraph images, the pre-event background brightness containing dust scattering (F-corona) and stray light is first subtracted from the respective polarizer images. To reduce noise, both the synthesized pB and tB images are smoothed using a median filter method (2×2 filter box).

[0087] Step 5-2: Obtain the three-dimensional CME area. track is the difference of CME relative time motion, and the convex hull method is used to obtain the preliminary CME region. Then, M smooth It is obtained by smoothing the edges of the CME region using a circular filter. The minimum FOV of COR1-A is subtracted from M to simulate the full extent of the CME structure.

[0088] Step 5-3: According to the preset mask M smooth , the position of the CME is calculated in three-dimensional space. The calculation results are expressed in the heliocentric-Earth equatorial (HEEQ) coordinate system, which has the center of the Sun as the origin, the Z axis aligned with the Sun's rotation axis, and the X axis located in a plane containing the Z axis and the Earth. It should be noted that the reconstruction results of the polarization are affected by the uncertainty of Thomson scattering. Polarization measurements can be used to derive the average distance of the CME from the plane of the sky, but it is impossible to determine on which side of the plane of the sky the CME occurs. However, the source region of the CME can be observed through ultraviolet, extreme ultraviolet or magnetic field data to resolve this problem.

[0089] In another embodiment, the present invention proposes a system for segmenting CME based on deep learning and automatically tracking and reconstructing CME in three dimensions, corresponding to the method for segmenting CME based on deep learning and automatically tracking and reconstructing CME in three dimensions proposed in the previous embodiment. The system includes:

[0090] The image preprocessing module is used to collect the observation data of the white-light coronagraph, preprocess the observation data, and obtain the corona differential image;

[0091] The image classification module is used to pre-train the VGG convolutional neural network model, perform binary classification on the coronal difference images, infer whether a single coronal difference image contains CME, and divide the preliminary CME events according to the rules of spatiotemporal continuity;

[0092] The image segmentation module is used to train the image segmentation model, perform pixel-by-pixel binary classification on the coronal difference image in the CME event, infer whether each pixel has a CME structure, and predict the CME structure area of ​​a single coronal difference image;

[0093] The CME tracking module defines tracking rules to track the CME structure evolution based on the prediction results of the CME structure area;

[0094] The 3D reconstruction module calculates the position distribution of CME in 3D space according to the tracking results based on the polarization ratio method.

[0095] The system processes a set of coronagraph observation data input by the user. First, preprocessing is performed to generate high-quality coronal differential images. Next, through four modules such as classification, segmentation, tracking and three-dimensional reconstruction, the refined CME area, structural evolution of CME events, related physical parameters and distribution in three-dimensional space can be obtained. The image classification module is used to determine whether there is a CME in the image, and the image segmentation module is responsible for extracting the structural area of ​​the CME. The CME tracking module tracks the structural evolution of a CME event and extracts related physical parameters, while the three-dimensional reconstruction module calculates the distribution of CME in three-dimensional space. The specific working principle and execution process of each module are the same as the method of segmenting CME based on deep learning and automatically tracking and reconstructing CME in three dimensions, so it will not be repeated here.

[0096] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A method for segmenting CME based on deep learning and automatically tracking and reconstructing CME in three dimensions, characterized in that: The steps include: Step 1: Collect the observation data of the white-light coronagraph, pre-process the observation data, and obtain the corona differential image; Step 2: Use the pre-trained convolutional neural network model to perform binary classification on the coronal difference images, infer whether a single coronal difference image contains CME, and divide the preliminary CME events according to the spatiotemporal continuity rule; Step 3: Train the image segmentation model to perform pixel-by-pixel binary classification on the coronal difference images in CME events, infer whether each pixel has a CME structure, and predict the CME structure area of ​​a single coronal difference image; Step 4: Based on the prediction results of the CME structure area, define tracking rules to track the CME structure evolution; Step 4 specifically includes the following sub-steps: Step 4-1: Perform image segmentation and post-processing on the image of the CME structure area; Step 4-2: Convert the processed image to the polar coordinate system, count the angular width, fit the velocity, select the CME structure area within the angular width range and the image that meets the velocity rule, and obtain the CME image sequence as the tracking result; Step 5: Based on the polarization ratio method, calculate the position distribution of the CME in three-dimensional space according to the tracking results; Step 5 specifically includes the following sub-steps: Step 5-1: Obtain polarization brightness and total brightness; Step 5-2: According to the tracking results, the CME area is obtained using the convex hull method, and then the edge of the CME area is smoothed using a circular filter to obtain the mask of the CME area; Step 5-3: Calculate the position of the CME in three-dimensional space by calculating the ratio of the polarized brightness to the total brightness in the mask.

2. The method for segmenting CME based on deep learning and automatically tracking and three-dimensionally reconstructing CME according to claim 1, characterized in that: In step 1, the coronal difference image is obtained by subtracting the previous frame from the current frame.

3. The method for segmenting CME based on deep learning and automatically tracking and three-dimensionally reconstructing CME according to claim 1, characterized in that: In step 2, the VGG convolutional neural network model is used to perform binary classification on the input coronal difference image based on the presence or absence of CME, and a predicted label of the coronal difference image is output, where a label of 1 indicates the presence of CME and a label of 0 indicates the absence of CME; based on the output predicted label, the input coronal difference image is divided into multiple CME image sequences in accordance with the spatiotemporal continuity rule to represent preliminary CME events.

4. The method for segmenting CME based on deep learning and automatically tracking and three-dimensionally reconstructing CME according to claim 3, characterized in that: Before dividing the input coronal difference image into multiple CME image sequences, the CME images containing two consecutive coronal difference images with labels of 0 are deleted; the time difference between each CME image sequence is calculated, and a time threshold is set for comparison, and according to the comparison result, it is determined whether the current CME image sequence is merged into the previous CME image sequence or the next CME image sequence or remains independent.

5. The method for segmenting CME based on deep learning and automatically tracking and three-dimensionally reconstructing CME according to claim 1, characterized in that: In step 3, the image segmentation model uses the Transformer-based semantic segmentation network model SegFormer, which uses the corona differential image in the CME event as input and outputs the CME structure area; the encoder of SegFormer consists of four different transformer blocks, which output four feature maps of different dimensions X i ,i∈1,2,3,4; the processing formula of SegFormer's decoder is as follows: X=MLP (4C,C) (Concat([X‘’1,X‘’2,X‘’3,X‘’4])) M=Upsample H×W (M p ) In the formula, MLP stands for multi-layer perceptron, the subscript represents dimension conversion, and C i represents the dimension of the i-th MLP, C represents the dimension unified by the linear layer, Upsample represents upsampling, Concat represents feature fusion, H and W represent the height and width of the input image respectively, and N class represents the number of categories, and M represents the CME structure area.

6. The method for segmenting CME based on deep learning and automatically tracking and reconstructing CME in three dimensions according to claim 1, characterized in that: In step 4-1, the image segmentation post-processing specifically includes: removing invalid and erroneous images, distinguishing image sequences based on temporal continuity, and distinguishing image sequences based on spatiotemporal continuity; The method of removing invalid and erroneous images specifically comprises: deleting the noise image, performing polar coordinate transformation on the remaining images, starting from the due north direction of the sun and sequentially unfolding in a counterclockwise direction, transforming each image at a resolution of θ×r, where θ is the polar angle and r is the radial distance measured from the center of the sun, counting the CME structure at each position angle, and discarding the image if the number of position angles containing the CME structure is less than a set value; The method of distinguishing image sequences based on time continuity specifically includes: for an image sequence exceeding a set number of frames, calculating the time difference between two adjacent images, and dividing the image sequence by comparing it with the set time value; The method of distinguishing image sequences based on spatiotemporal continuity is specifically as follows: converting an image sequence exceeding a set number of frames into a polar coordinate system, identifying the propagation angle of the CME structure by comparing the height difference between consecutive images, judging whether the CME structure propagates radially outward according to the propagation angle, marking the CME structure that propagates radially outward, and on this basis, using a marking morphological operator to process it into connected areas with different color marks, the boundaries of the same color areas are marked by bounding boxes, merging overlapping bounding boxes, and dividing the image sequence according to the bounding boxes.

7. The method for segmenting CME based on deep learning and automatically tracking and three-dimensionally reconstructing CME according to claim 1, characterized in that: In step 4-2, the statistical angular width is specifically: calculating the maximum height at each position angle, and merging the discrete position angles into one or more angular widths based on the height; the fitting speed includes the median speed, the maximum speed and the acceleration.

8. A system for segmenting CME based on deep learning and automatically tracking and reconstructing CME in three dimensions, characterized in that: include: The image preprocessing module is used to collect the observation data of the white-light coronagraph, preprocess the observation data, and obtain the corona differential image; The image classification module is used to pre-train the VGG convolutional neural network model, perform binary classification on the coronal difference images, infer whether a single coronal difference image contains CME, and divide the preliminary CME events according to the rules of spatiotemporal continuity; The image segmentation module is used to train the image segmentation model, perform pixel-by-pixel binary classification on the coronal difference image in the CME event, infer whether each pixel has a CME structure, and predict the CME structure area of ​​a single coronal difference image; The CME tracking module defines tracking rules to track the evolution of CME structures based on the prediction results of the CME structure area. It includes: performing image segmentation and post-processing on the images of the CME structure area; converting the processed images to the polar coordinate system, counting the angular width, fitting the velocity, selecting the CME structure area within the angular width range and the images that meet the velocity rules, and obtaining the CME image sequence as the tracking result; The three-dimensional reconstruction module calculates the position distribution of CME in three-dimensional space based on the tracking results based on the polarization ratio method; it includes: obtaining the polarization brightness and total brightness; obtaining the CME area based on the tracking results using the convex hull method, and then using a circular filter to smooth the edge of the CME area to obtain the mask of the CME area; calculating the position of CME in three-dimensional space by calculating the ratio of the polarization brightness and the total brightness in the mask.

Images