All-sky aurora video unsupervised event segmentation method based on bidirectional perception

Abstract

The invention relates to an all-sky aurora video unsupervised event segmentation method based on bidirectional perception, which comprises the following steps of: 1, inputting an aurora image sequence, randomly selecting a target frame, forming two subsequences which take the target frame as a center by using the front frame and the rear frame of the target frame as positive samples, and enabling the features of the target frame to have context information; step 2, carrying out comparative learning on the extracted target frame features to obtain a higher-level representation, masking the target frame, then carrying out bidirectional feature reconstruction, reconstructing the target frame, and training a reconstruction model; 3, testing the trained reconstruction model, performing reconstruction and similarity calculation on the aurora image sequence in the test set frame by frame to obtain an error curve, regarding the error curve as a boundary probability curve, performing smoothing processing by using a filter, setting a boundary probability threshold, and regarding a frame which is greater than the boundary probability threshold and is a peak value as a boundary frame; the method has the characteristic of high aurora sequence segmentation accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an unsupervised event segmentation method for all-sky aurora videos based on bidirectional perception. Background Art

[0002] In the field of machine learning, while traditional supervised learning techniques have achieved great success in various applications, they rely on large amounts of labeled data, which is both costly and time-consuming. Supervised learning's heavy reliance on manually labeled data limits its scalability to a wider range of application scenarios. Consequently, unsupervised learning has become increasingly important. It can eliminate the need for data labeling by learning the inherent structure and patterns of unlabeled data, reducing both the need for data labeling and the cost. The advantages of unsupervised learning lie not only in cost savings but also in its adaptability to a wider range of applications, particularly when labeled data is difficult to obtain or incomplete. For example, unsupervised learning methods can be used for a variety of tasks, such as clustering, anomaly detection, and density estimation. These techniques enable models to automatically discover correlations and patterns in datasets without external intervention. Furthermore, unsupervised learning makes it possible to process large datasets by learning directly from raw data without requiring extensive preprocessing or labeling. This allows machine learning models to be more flexibly applied to a variety of data environments, particularly those that are dynamically changing and continuously evolving.

[0003] The emergence of self-supervised learning has injected new vitality into unsupervised learning. It requires models to be trained solely on the data itself, without the need for additional labeled information. Within the field of self-supervised learning, contrastive learning strategies have garnered significant attention in recent years. This strategy aims to cluster similar samples together while pushing dissimilar samples apart, enabling the model to learn the essential characteristics of the data and thus better distinguish samples in the feature space. The core idea of ​​contrastive learning is to learn representations based on instance-based discrimination by analyzing the similarities and differences between samples. This approach has not only been widely used in vision tasks such as image retrieval, feature learning, and object detection, but has also played a crucial role in the further development of self-supervised learning. Its simple and intuitive concept and outstanding performance have made it a popular choice among researchers and an indispensable component of self-supervised learning. Through contrastive learning, models can learn useful feature representations from large amounts of unlabeled data. These representations can generalize to multiple tasks and, in many cases, are more robust than traditional supervised learning methods. In practical applications, contrastive learning has proven to be a powerful and effective unsupervised learning method, providing powerful tools and technical support for solving various real-world machine learning problems.

[0004] The main problems in the Aurora sequence segmentation task are as follows:

[0005] 1) Because the aurora's structure is non-rigid, lacking clear boundaries and shapes, its motion involves numerous transition states, and its evolution time varies, with no fixed length. Therefore, the sequences that need to be processed are often of varying lengths. This poses significant challenges to both extracting auroral features and segmenting auroral image sequences.

[0006] 2) The spatial directionality of auroral events is closely linked to their temporal order. Existing models focus too narrowly on local information, neglecting the temporal order that captures the dynamics of auroras. This can lead to reduced accuracy in identifying event boundaries.

[0007] In summary, due to the lack of clear boundaries in aurora sequences, existing models perform far worse on aurora sequence segmentation tasks than on datasets with more distinct boundaries. Therefore, a method that simultaneously maintains a global perspective and directional learning is needed to improve the accuracy of aurora sequence segmentation. Summary of the Invention

[0008] In order to overcome the deficiencies of the above-mentioned prior art, the object of the present invention is to provide an unsupervised event segmentation method for all-sky aurora videos based on bidirectional perception, which has the characteristic of high accuracy in aurora sequence segmentation.

[0009] To achieve the above objectives, the present invention adopts a technical solution: an unsupervised event segmentation method for all-sky aurora videos based on bidirectional perception, comprising the following steps:

[0010] Step 1: Input a set of full-sky aurora image sequences. After intensity adjustment and preprocessing (dark current removal, cropping, and rotation), the feature extractor extracts the shallow sequence features of the aurora image sequence. A frame is randomly selected from the shallow sequence features as the target frame. The target frame and the T-1 frames before the target frame are selected to form a forward subsequence centered on the target frame. The target frame and the T-1 frames after the target frame are selected to form a backward subsequence centered on the target frame. In order to capture the high-level feature representation of the aurora image sequence, contrastive learning is introduced for the forward subsequence and the backward subsequence. The specific implementation is as follows: For the forward subsequence, the T-1 frames before the target frame are selected as positive samples; for the backward subsequence, the T-1 frames after the target frame are selected as positive samples. A multi-level negative sample selection strategy is also designed, including negative sample pairs with a large time span within the same video (intra-video negative pairs), cross-video negative sample pairs between different videos (inter-video negative pairs), and negative sample pairs embedded in memory during iteration (memory negative pairs). Finally, cosine similarity is used to measure similarity. Based on the similarity results, a contrastive loss function is designed to bring positive samples with high similarity closer and negative samples with low similarity away.

[0011] Step 2: Based on the forward subsequence and backward subsequence features in step 1, a bidirectional feature reconstruction module is designed to reconstruct the target frame. The specific implementation is as follows: first, position encoding is added to the forward subsequence and backward subsequence after contrastive learning, and then a mask operation is performed on the target frames of the forward subsequence and backward subsequence. Finally, the modules are input into a reconstruction network composed of a multi-layer Transformer to obtain the forward target frame reconstructed from the forward subsequence and the backward target frame reconstructed from the backward subsequence. Minimizing the mean square error is used as the loss function, and the model training is supervised by calculating the pixel-level difference between the reconstructed forward target frame, the reconstructed backward target frame and the original target frame to obtain a trained bidirectional feature reconstruction model.

[0012] Step 3, test the reconstruction model of step 2 to obtain the boundary probability curve; the specific implementation is: the reconstruction model sequentially scans the aurora image sequence from the beginning to the end, reconstructs the frame features frame by frame, obtains the reconstruction error of each frame, and forms an error curve; according to the continuity of the event, the error between the reconstructed forward target frame and the reconstructed backward target frame and the original target frame should be small, that is, the frame with a large reconstruction error indicates a high probability of belonging to the boundary frame, so the error curve is regarded as the boundary probability curve; the boundary probability curve is smoothed using a mean filter and a high-pass filter, and a boundary probability threshold is set. The frame that is greater than the boundary probability threshold and is a peak is regarded as a boundary frame.

[0013] The comparative study of the full-sky aurora image sequence described in step 1 is performed as follows:

[0014] Step 1-1: Perform pre-processing operations on the full-sky aurora image sequence by intensity stretching, dark current subtraction, and cropping and rotation, and then perform shallow feature extraction to obtain shallow features X∈R B×C×H×W ;

[0015] Step 1-2: Randomly select a frame from the sequence features to be segmented as the target frame, and select its previous T-1 frames to form a forward subsequence H1{x t-(T-1) ,...,x t-1 ,x t}, select the next T-1 frames to form a backward subsequence H2{x t ,x t+1 ,...,x t+(T-1)};

[0016] Step 1-3: For the forward subsequence, select the T-1 frames before the target frame as positive samples; for the backward subsequence, select the T-1 frames after the target frame as positive samples; there are three sources of negative samples, namely intra-video negative pairs (negative samples come from the same aurora image sequence but from different clips), inter-video negative pairs (negative samples are selected from any clips extracted from different aurora sequences) and memory negative pairs (negative samples come from frames embedded in the memory during the previous iteration);

[0017] Steps 1-4: Use cosine similarity to measure similarity. Based on the similarity results, design a loss function to bring samples with high similarity closer and move away from samples with low similarity. The selected loss function is InfoNCE Loss in contrastive learning. Contrastive learning uses the similarity score of the positive sample pair as the numerator and the similarity scores of all positive and negative sample pairs as the denominator to calculate its negative log-likelihood loss, increasing the similarity score of the positive sample pair and decreasing the similarity score of the negative sample pair. See the formula as shown:

[0018]

[0019] in, Represents the contrast loss obtained by feature sequence H1 and feature sequence H2, represents the selection of the smallest value between the two for back propagation, q represents the query item, k i represents all sample keys, τ is the temperature parameter, k + is the positive sample key.

[0020] The target frame reconstruction described in step 2 includes the following steps:

[0021] Step 2-1: Position encoding is performed on the two input subsequences, namely the forward subsequence and the backward subsequence. The position encoding is combined with the content encoding of the sequence to comprehensively represent the information of each position in the sequence; masking operations are performed on the two subsequences to obtain two masked subsequences; for the forward subsequence: the target frame is selected at the end of the forward subsequence, and the masking operation is performed on the tail frame to hide the information of the tail frame, so that the bidirectional feature reconstruction module cannot obtain the information of the tail frame; for the backward subsequence: the target frame is selected at the beginning of the backward subsequence, and the masking operation is performed on the first frame to hide the information of the first frame, so that the bidirectional feature reconstruction module cannot obtain the information of the first frame; the position encoding formula after masking is added as follows:

[0022] H′1={h t-(T-1) ,...,h t-1 ,[MASK] t}+{pos t-(T-1) ,...,pos t-1 ,[MASK] t}

[0023] H′2={[MASK] t ,h t+1 ,....,h t+(T-1)}+{[MASK] t ,pos t+1 ,...,pos t+(T-1)}

[0024] Among them, H1' represents masking and adding position coding to the last frame of the forward subsequence, H2' represents masking and adding position coding to the first frame of the backward subsequence, MASK represents masking, and pos represents position coding;

[0025] Step 2-2: The two subsequences masked in step 2-1 are input into a reconstruction network consisting of a multi-layer Transformer decoder. The forward reconstruction part is responsible for forward reconstruction of the masked tail frame. By learning the relationship between the previous frame and the tail frame, the bidirectional feature reconstruction module reconstructs the tail frame. The backward part is responsible for backward reconstruction of the first frame. By learning the relationship between the subsequent frames and the first frame, the bidirectional feature reconstruction module reconstructs the first frame.

[0026] Step 2-3: By balancing the two reconstruction losses and the loss between the original features, the constraints on the bidirectional reconstruction module are realized, and the mean square error loss is used to measure the reconstructed features, which is divided into bidirectional reconstruction loss and the original reconstruction loss The formula is:

[0027]

[0028] in represents the mean squared loss of the forward target frame and the backward target frame, Represents the minimum value of the mean square loss between the forward target frame and the original frame and the mean square loss between the backward target frame and the original frame, h t1 represents the forward reconstruction feature, h t2 represents the backward reconstruction feature, h t Represents the original frame features, s is the number of sequence frames, and V is the number of videos.

[0029] The boundary probability curve described in step 3 is obtained by:

[0030] Step 3-1: Replace the dataset of the trained bidirectional feature reconstruction model with the test set, sequentially scan the aurora image sequence from the beginning to the end, reconstruct the frame features of the target frame, obtain the reconstruction error of each frame, and form an error curve; for the same event, due to the continuity of the event, the error between the reconstructed target frame and the original target frame is small, that is, the frame with a large reconstruction error indicates a high probability of belonging to the boundary frame, so the error curve is regarded as the boundary probability curve;

[0031] Step 3-2: Using a moving average filter, starting from the beginning of the signal sequence, select a data window of length N and calculate the average value of the data in the data window as the output at the current moment. Then, slide the data window backward by one data point and repeat the above calculation process until the entire signal sequence is processed. Use an FIR high-pass filter to remove the low-frequency noise boundary and retain the peak at the high-frequency event boundary.

[0032] Step 3-3: Calculate the gradient of the reconstruction error trajectory to obtain the gradient trajectory, which reflects the rate of change of the reconstruction error; detect the relative maximum value on the gradient trajectory, and the relative extreme value points are regarded as potential boundary positions because they mean that the reconstruction error has undergone significant local changes near this point; finally, perform threshold judgment on the detected relative extreme value points to distinguish which are true event boundaries and which are extreme values ​​caused by noise or local changes. Reasonable threshold screening can effectively improve the accuracy of boundary recognition.

[0033] Compared with the prior art, the present invention has the following significant advantages:

[0034] Efficient unsupervised learning framework: It completely gets rid of the dependence on labeled data, avoids the high labeling costs and long cycle required by traditional supervised learning methods in aurora image sequence segmentation, and significantly reduces data preprocessing costs.

[0035] Accurate auroral event segmentation: Through a bidirectional feature reconstruction mechanism based on contrastive learning, this method effectively captures the dynamic characteristics of the auroral non-rigid structure and achieves high-precision boundary detection. Experimental verification shows that it outperforms existing unsupervised methods in metrics such as TPR (90.13%), F-measure (87.82%), and similarity (78.29%).

[0036] Advanced sequence modeling capabilities: Utilizing a bidirectional perception mechanism (front and back subsequence reconstruction + contrastive learning), it overcomes the limitations of traditional models in capturing local information, significantly improves the modeling capabilities of long-term auroral motion, and enhances adaptability to dynamic evolution processes.

[0037] Robust multi-level negative sample optimization: A triple negative sample mechanism consisting of intra-video negative pairs, cross-video negative pairs, and memory negative pairs is adopted to enhance the discrimination of contrastive learning and improve the generalization ability of feature representation, enabling the model to better adapt to the differences between different aurora sequences.

[0038] Adaptive parameter optimization: By optimizing the configuration of filters and extreme value detection (such as filter size, subsequence length, etc.), it effectively balances the sensitivity of noise suppression and peak detection, avoiding the problems of missed detection or misjudgment caused by fixed parameters in traditional methods.

[0039] The present invention solves the defect that the aurora image sequence cannot be accurately segmented because the aurora image has no clear boundary and shape, resulting in unclear boundary frames.

[0040] Key data support:

[0041] Compared with the supervised method PC (TPR 88.81%), the recall rate of the proposed method (90.13%) is improved by 1.32%;

[0042] Compared with the best unsupervised method (spatiotemporal low-rank attention + CoSeg), F-measure increased from 84.85% to 87.82%;

[0043] The soft true positive rate (STP) within the boundary detection error tolerance interval (±2 minutes) is over 90%;

[0044] This technology has been verified by actual data from aurora observation stations, providing a reliable analysis tool for intelligent space weather monitoring and aurora physics research.

[0045] The present invention is particularly suitable for fields requiring high-precision time series analysis, such as auroral substorm event monitoring and space weather forecasting, and provides innovative technical support for the automated processing of astronomical observation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Flowchart of the method of the present invention.

[0047] Figure 2 This is a visualization result diagram of the method of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0049] Example 1

[0050] See also Figure 1 , an unsupervised event segmentation method for all-sky aurora videos based on bidirectional perception, comprising the following steps:

[0051] Step 1: Input a set of full-sky aurora image sequences. After intensity adjustment and preprocessing (dark current removal, cropping, and rotation), the feature extractor extracts the shallow sequence features of the aurora image sequence. A frame is randomly selected from the shallow sequence features as the target frame. The target frame and the T-1 frames before the target frame are selected to form a forward subsequence centered on the target frame. The target frame and the T-1 frames after the target frame are selected to form a backward subsequence centered on the target frame. In order to capture the high-level feature representation of the aurora image sequence, contrastive learning is introduced for the forward subsequence and the backward subsequence. The specific implementation is as follows: For the forward subsequence, the T-1 frames before the target frame are selected as positive samples; for the backward subsequence, the T-1 frames after the target frame are selected as positive samples. A multi-level negative sample selection strategy is also designed, including negative sample pairs with a large time span within the same video (intra-video negative pairs), cross-video negative sample pairs between different videos (inter-video negative pairs), and negative sample pairs embedded in memory during iteration (memory negative pairs). Finally, cosine similarity is used to measure similarity. Based on the similarity results, a contrastive loss function is designed to bring positive samples with high similarity closer and negative samples with low similarity away.

[0052] Step 2: Based on the forward subsequence and backward subsequence features in step 1, a bidirectional feature reconstruction module is designed to reconstruct the target frame. The specific implementation is as follows: first, position encoding is added to the forward subsequence and backward subsequence after contrastive learning, and then a mask operation is performed on the target frames of the forward subsequence and backward subsequence. Finally, the modules are input into a reconstruction network composed of a multi-layer Transformer to obtain the forward target frame reconstructed from the forward subsequence and the backward target frame reconstructed from the backward subsequence. Minimizing the mean square error is used as the loss function, and the model training is supervised by calculating the pixel-level difference between the reconstructed forward target frame, the reconstructed backward target frame and the original target frame to obtain a trained bidirectional feature reconstruction model.

[0053] Step 3, test the reconstruction model of step 2 to obtain the boundary probability curve; the specific implementation is: the reconstruction model sequentially scans the aurora image sequence from the beginning to the end, reconstructs the frame features frame by frame, obtains the reconstruction error of each frame, and forms an error curve; according to the continuity of the event, the error between the reconstructed forward target frame and the reconstructed backward target frame and the original target frame should be small, that is, the frame with a large reconstruction error indicates a high probability of belonging to the boundary frame, so the error curve is regarded as the boundary probability curve; the boundary probability curve is smoothed using a mean filter and a high-pass filter, and a boundary probability threshold is set. The frame that is greater than the boundary probability threshold and is a peak is regarded as a boundary frame.

[0054] The comparative study of the full-sky aurora image sequence described in step 1 is performed as follows:

[0055] Step 1-1: Perform pre-processing operations on the full-sky aurora image sequence by intensity stretching, dark current subtraction, and cropping and rotation, and then perform shallow feature extraction to obtain shallow features X∈R B×C×H×W ;

[0056] Step 1-2: Randomly select a frame from the sequence features to be segmented as the target frame, and select its previous T-1 frames to form a forward subsequence H1{x t-(T-1) ,...,x t-1 ,x t}, select the next T-1 frames to form a backward subsequence H2{x t ,x t+1 ,...,x t+(T-1)};

[0057] Step 1-3: For the forward subsequence, select the T-1 frames before the target frame as positive samples; for the backward subsequence, select the T-1 frames after the target frame as positive samples; there are three sources of negative samples, namely intra-video negative pairs (negative samples come from the same aurora image sequence but from different clips), inter-video negative pairs (negative samples are selected from any clips extracted from different aurora sequences) and memory negative pairs (negative samples come from frames embedded in the memory during the previous iteration);

[0058] Steps 1-4: Use cosine similarity to measure similarity. Based on the similarity results, design a loss function to bring samples with high similarity closer and move away from samples with low similarity. The selected loss function is InfoNCE Loss in contrastive learning. Contrastive learning uses the similarity score of the positive sample pair as the numerator and the similarity scores of all positive and negative sample pairs as the denominator to calculate its negative log-likelihood loss, increasing the similarity score of the positive sample pair and decreasing the similarity score of the negative sample pair. See the formula as shown:

[0059]

[0060] in, Represents the contrast loss obtained by feature sequence H1 and feature sequence H2, represents the selection of the smallest value between the two for back propagation, q represents the query item, k i represents all sample keys, τ is the temperature parameter, k + is the positive sample key.

[0061] The target frame reconstruction described in step 2 includes the following steps:

[0062] Step 2-1: Position encoding is performed on the two input subsequences, namely the forward subsequence and the backward subsequence. The position encoding is combined with the content encoding of the sequence to comprehensively represent the information of each position in the sequence; masking operations are performed on the two subsequences to obtain two masked subsequences; for the forward subsequence: the target frame is selected at the end of the forward subsequence, and the masking operation is performed on the tail frame to hide the information of the tail frame, so that the bidirectional feature reconstruction module cannot obtain the information of the tail frame; for the backward subsequence: the target frame is selected at the beginning of the backward subsequence, and the masking operation is performed on the first frame to hide the information of the first frame, so that the bidirectional feature reconstruction module cannot obtain the information of the first frame; the position encoding formula after masking is added as follows:

[0063] H1′={h t-(T-1) ,...,h t-1 ,[MASK] t}+{pos t-(T-1) ,...,pos t-1 ,[MASK] t}

[0064] H2′={[MASK] t ,h t+1 ,....,h t+(T-1)}+{[MASK] t ,pos t+1 ,...,pos t+(T-1)}

[0065] Among them, H1' represents masking and adding position coding to the last frame of the forward subsequence, H2' represents masking and adding position coding to the first frame of the backward subsequence, MASK represents masking, and pos represents position coding;

[0066] Step 2-2: The two subsequences masked in step 2-1 are input into a reconstruction network consisting of a multi-layer Transformer decoder. The forward reconstruction part is responsible for forward reconstruction of the masked tail frame. By learning the relationship between the previous frame and the tail frame, the bidirectional feature reconstruction module reconstructs the tail frame. The backward part is responsible for backward reconstruction of the first frame. By learning the relationship between the subsequent frames and the first frame, the bidirectional feature reconstruction module reconstructs the first frame.

[0067] Step 2-3: By balancing the two reconstruction losses and the loss between the original features, the constraints on the bidirectional reconstruction module are realized, and the mean square error loss is used to measure the reconstructed features, which is divided into bidirectional reconstruction loss and the original reconstruction loss The formula is:

[0068]

[0069] in represents the mean squared loss of the forward target frame and the backward target frame, Represents the minimum value of the mean square loss between the forward target frame and the original frame and the mean square loss between the backward target frame and the original frame, h t1 represents the forward reconstruction feature, h t2 represents the backward reconstruction feature, h t Represents the original frame features, s is the number of sequence frames, and V is the number of videos.

[0070] The boundary probability curve described in step 3 is obtained by:

[0071] Step 3-1: Replace the trained reconstruction model dataset with the test dataset, sequentially scan the aurora image sequence from the beginning to the end, reconstruct the frame features of the target frame, obtain the reconstruction error of each frame, and form an error curve. For the same event, based on the continuity of the event, the error between the reconstructed target frame and the original target frame is small, that is, the frame with a large reconstruction error indicates a high probability of belonging to the boundary frame. Therefore, the error curve is regarded as the boundary probability curve;

[0072] Step 3-2: Using a moving average filter, starting from the beginning of the signal sequence, select a data window of length N and calculate the average value of the data in the data window as the output at the current moment. Then, slide the data window backward by one data point and repeat the above calculation process until the entire signal sequence is processed. Use an FIR high-pass filter to remove the low-frequency noise boundary and retain the peak at the high-frequency event boundary.

[0073] Step 3-3: Calculate the gradient of the reconstruction error trajectory to obtain the gradient trajectory, which can reflect the rate of change of the reconstruction error; detect the relative maximum value on the gradient trajectory, and the relative extreme value points are regarded as potential boundary positions because they mean that the reconstruction error has undergone significant local changes near this point; finally, perform threshold judgment on the detected relative extreme value points to distinguish which are true event boundaries and which are extreme values ​​caused by noise or local changes. Reasonable threshold screening can effectively improve the accuracy of boundary recognition.

[0074] Example 2

[0075] An unsupervised event segmentation method for all-sky aurora videos based on bidirectional perception includes the following steps:

[0076] 1) A full-sky auroral image sequence is input. After intensity adjustment and preprocessing (dark current removal, cropping, and rotation), the feature extractor extracts shallow features from the auroral image sequence. A frame is randomly selected from the sequence features to be segmented as the target frame. The T-1 frames preceding it are used to form a forward subsequence centered on the target frame, and the T-1 frames following it are used to form a backward subsequence centered on the target frame. To capture high-level feature representations of the auroral image sequence, contrastive learning is introduced for the forward and backward subsequences. Specifically, for the forward subsequence, the T-1 frames preceding the target frame are used as positive samples; for the backward subsequence, the T-1 frames following the target frame are used as positive samples. A multi-level negative sample selection strategy is also designed, including negative sample pairs with a large temporal span within the same video (intra-video negative pairs), cross-video negative sample pairs between different videos (inter-video negative pairs), and negative sample pairs selected from a global feature library (memory negative pairs). Finally, cosine similarity is used to measure similarity. Based on the similarity results, a contrastive loss function is designed to bring samples with high similarity closer and move away from samples with low similarity. The loss function uses the InfoNCE Loss of contrastive learning, as shown in the following formula:

[0077]

[0078] in, Represents the contrast loss obtained by feature sequence H1 and feature sequence H2, represents the selection of the smallest value between the two for back propagation, q represents the query item, k i represents all sample keys, τ is the temperature parameter, k + is the positive sample key.

[0079] 2) After expanding the field of view of the target frame through contrastive learning, a bidirectional global feature reconstruction is then designed to ensure that the model can accurately connect the events that occur one after another in the sequence by learning directional information, thereby capturing the long-term motion trend of the aurora in the sequence. Specifically, the two feature sequences H1 and H2 are combined with their respective position encodings. Subsequently, for both sequences, for any selected frame, if it is at the end of the sequence, the tail frame is blocked; similarly, if the selected frame is at the beginning of the sequence, the first frame is blocked. The two masked sequences are input into the reconstruction module. That is, the following is obtained:

[0080] H1′={h t-(T-1) ,...,h t-1 ,[MASK] t}+{pos t-(T-1) ,...,pos t-1 ,[MASK] t}

[0081] H2′={[MASK] t ,h t+1 ,....,h t+(T-1)}+{[MASK] t ,pos t+1 ,...,pos t+(T-1)}

[0082] Among them, H1' represents masking and adding position coding to the last frame of the forward subsequence, H2' represents masking and adding position coding to the first frame of the backward subsequence, MASK represents masking, and pos represents position coding;

[0083] H1 reconstructs the mask frame from the positive sequence direction by analyzing the semantic information of the unmasked first T-1 frames in the sequence to obtain the reconstructed feature h t1 ; H2′ reconstructs the frame in reverse order through the T-1 frame following the frame to obtain the reconstructed feature h t2 The reconstructed features are then measured using the mean square error loss, which is divided into a bidirectional reconstruction loss. and the original reconstruction loss The formula is as follows:

[0084]

[0085] in represents the mean squared loss of the forward target frame and the backward target frame, Represents the minimum value of the mean square loss between the forward target frame and the original frame and the mean square loss between the backward target frame and the original frame, h t1 represents the forward reconstruction feature, h t2 represents the backward reconstruction feature, ht Represents the original frame features, s is the number of sequence frames, and V is the number of videos.

[0086] 3) After the model training is completed, the aurora image sequence in the test set is tested. First, we reconstruct each frame of the image and calculate its similarity with the original image to obtain an error curve. This error curve reflects the difference between the reconstructed image and the original image, and we regard it as a boundary probability curve. On this basis, in order to eliminate the noise caused by error fluctuations, we use a filter to smooth the error curve. The smoothed curve can more accurately reflect the changing trend of the boundary during the image reconstruction process. Next, a boundary probability threshold is set, and points greater than the threshold are regarded as potential boundary frames. In addition, in order to further improve the discrimination accuracy, we also filter the peaks in the boundary probability curve, and only frames above the threshold and located at the peak position are considered boundary frames.

[0087] Example 3

[0088] An unsupervised event segmentation method for all-sky aurora videos based on bidirectional perception includes the following steps:

[0089] Step 1: The CCD imager continuously captures the entire sky of the aurora at 557.7 nm. The 557.7 nm image acquisition process involves capturing an image every 10 seconds. After the data is read, it is converted into an aurora image sequence, intensity adjusted, and preprocessed (dark current removal, cropping, and rotation).

[0090] Step 2: The feature extractor extracts sequence features. It randomly selects a target frame from the sequence to be segmented as the center. It then selects the T-1 frames before and after it to form two subsequences centered on the target frame. The forward subsequence and the backward subsequence are contrastively learned to obtain a higher-level feature representation. The target frames of the two subsequences are masked and reconstructed to obtain the forward reconstructed target frame and the backward reconstructed target frame. A mean square loss is applied between the two reconstructed target frames, and a mean square loss is also applied between the forward reconstructed target frame and the original target frame. The model is trained by minimizing the contrast loss and reconstruction loss.

[0091] 2.1) The feature extractor extracts shallow sequence features. It randomly selects a target frame from the sequence to be segmented as the center, and selects the T-1 frames before and after it to form two subsequences centered on the target frame. The two subsequences are:

[0092] H1:{x t-(T-1),..., x t-1, x t} and H2:{x t, x t+1,..., x t+(T-1)}

[0093] 2.2) Positive sample selection for contrastive learning: For the forward subsequence, the T-1 frames before the target frame are selected as positive samples; for the backward subsequence, the T-1 frames after the target frame are selected as positive samples;

[0094] 2.3) Contrastive learning negative sample selection: intra-video negative pairs (negative samples come from the same aurora image sequence but from different segments), inter-video negative pairs (negative samples are selected from any segments extracted from different aurora sequences), and memory negative pairs (negative samples come from frames embedded in the memory during the previous iteration);

[0095] 2.4) Contrastive Learning: Use cosine similarity to measure the similarity between two subsequences. Based on the similarity results, a loss function is designed to bring samples with high similarity closer and samples with low similarity further away. The loss function selected here is the InfoNCE loss commonly used in contrastive learning. Specifically, the model uses the similarity score of the positive sample pair as the numerator and the similarity scores of all positive and negative sample pairs as the denominator to calculate its negative log-likelihood loss. Among them, the contrast loss obtained by the feature sequence H1 is denoted as The contrast loss obtained by the feature sequence H2 is recorded as Select the smallest value of the two for back propagation, the formula is:

[0096]

[0097] 2.5) Bidirectional Feature Reconstruction: Positional encoding of the input sequence introduces positional information, helping the model distinguish content at different positions and better understand sequential relationships within the sequence. Positional encoding is combined with content encoding. Positional embedding uses sine and cosine functions to calculate encoding values ​​for different positions, taking into account the relative distance between positions. Masking the two subsequences allows the model to consider only future or past information, thereby better capturing motion information within the sequence. Specifically, masking the two sequences follows the following steps: For the first sequence, the target frame is selected at the end of the sequence and masked, thereby hiding the information of the last frame and preventing the model from accessing it. For the second sequence, the target frame is selected at the beginning of the sequence and masked, thereby hiding the information of the first frame and preventing the model from accessing it. The two masked subsequences are input into the reconstruction module. The forward part is responsible for reconstructing the masked previous frame. By learning the relationship between the previous and current frames, the model can reconstruct the current frame. The reverse part is responsible for reconstructing the subsequent frames in reverse order. By learning the relationship between the subsequent frames and the current frame, the model can reconstruct the current frame. The mean square error loss is used to measure the reconstructed features, which is divided into two-way reconstruction loss. and the original reconstruction loss The bidirectional reconstruction module is constrained by balancing the two reconstruction losses and the loss with the original features;

[0098] 2.6) The loss function is: Use the backpropagation algorithm and optimizer to update the model parameters to minimize the loss function. The optimizer selected here is SGD;

[0099] Step 3: Test the trained model. Reconstruct the aurora image sequence in the test set frame by frame and calculate the similarity to obtain the error curve, which is regarded as the boundary probability curve. Use a filter to smooth it, set the boundary probability threshold, and regard the frames with peak values ​​greater than the boundary probability threshold as boundary frames.

[0100] 3.1) Replace the trained model's dataset with the test set, sequentially scan the aurora image sequence from beginning to end, reconstruct the target frame's frame features, and calculate the reconstruction error for each frame to form an error curve. If the event is the same, then due to the continuity of the event, the reconstructed target frame will be very close to the original target. In other words, frames with large errors have a high probability of being boundary frames, so the error curve is considered a boundary probability curve.

[0101] 3.2) Smoothing: Using a moving average filter, starting from the beginning of the signal sequence, select a data window of length N and calculate the average value of the data within the window as the output at the current moment. The window is then moved back one data point at a time, and the above calculation process is repeated until the entire sequence is processed. An FIR high-pass filter is used to remove the low-frequency noise boundary and retain the peaks at the high-frequency event boundary.

[0102] 3.3) Relative Extreme Value Detection: The gradient of the reconstruction error trajectory is calculated to obtain a gradient trajectory that reflects the rate of change of the reconstruction error. Relative maxima on the gradient trajectory are detected. These relative extreme value points can be considered potential boundary locations because they indicate significant local changes in the reconstruction error near that point. Finally, a threshold judgment is performed on the detected relative extreme value points to distinguish between true event boundaries and extreme values ​​caused by noise or local changes. Appropriate threshold screening can effectively improve the accuracy of boundary identification.

[0103] After detecting the boundary frame, it is necessary to judge the effectiveness of the model testing phase. The specific judgment steps are as follows:

[0104] In step S1, in order to more accurately quantify the dynamically changing aurora image sequence, the following soft indicators are further set to evaluate the performance of the model:

[0105] Soft True Positive (STP): If the detected boundary is within the time interval of 2 minutes before and after the marked event segmentation line, it is considered a soft true positive;

[0106] Soft False Positive (SFP): The detected boundary is located in a time period that is not related to the real event and is not within the time range of 2 minutes before and after the annotated segmentation line. This indicates that the model has incorrectly marked a change in a non-existent event;

[0107] Soft False Negative (SFN): If the expected peak does not appear in the 2-minute time interval before and after the labeled event, it is a soft false negative. This means that the model failed to correctly detect the actual event.

[0108] In step S2, the True Positive Rate (TPR), i.e., recall rate, precision, F-Measure(), and similarity index are used as evaluation criteria to prove the effectiveness of the proposed method, namely:

[0109]

[0110] In order to maximize the performance of the model, it is necessary to configure various parameters. The specific configuration steps are as follows:

[0111] In step n1, the key to effectively segmenting auroral image sequences lies in precisely adjusting and selecting appropriate filter parameters, subsequence length (T), and extreme value detection range (r). To this end, a series of experiments were designed to determine the optimal parameter combination to improve the accuracy and reliability of auroral image sequence segmentation.

[0112] When smoothing the bidirectional reconstruction error, a filter size that is too small may result in excessive burrs in the curve, failing to highlight the most important peaks. However, a filter size that is too large may result in an overly smooth curve, smoothing out peaks that should be detected and causing them to be missed. Therefore, the following experiment was conducted to determine the filter size that is most suitable for segmenting aurora image sequences. Let k1 be the size of filter 1 and k2 be the size of filter 2. When k1 = 60 and k2 = 30, the optimal balance between curve smoothness and peak visibility is achieved, resulting in the most accurate edge detection.

[0113]

[0114] In step n2, the present invention further tested the effects of subsequence length T and extreme value detection range r. A longer subsequence length provides a wider field of view, but may result in loss of local information. On the other hand, an excessively large extreme value detection range may overlook small-scale local auroral events, while an excessively small range may overemphasize noise on the curve, thus affecting the accuracy of the detection results. Therefore, as shown in the table below, experiments were conducted on the parameter values ​​of T and r, ultimately selecting T = 10 and r = 40 as the optimal parameter combination.

[0115]

[0116] In order to fully demonstrate the usefulness of the present invention, the present invention conducted a series of experiments and compared with other models, as shown in the table below. Through the comparative experimental results with other methods, it can be seen that the present invention has shown more significant advantages in indicators such as positive rate (TPR), accuracy, F1 and similarity in the unsupervised field; although the accuracy has decreased compared with the supervised method, other indicators have advantages. This shows that the present invention can better detect the boundaries of auroral events and segment long auroral image sequences into units containing individual auroral events, which shows the efficiency and comprehensiveness of the algorithm. Compared with traditional methods, the present invention can better capture the dynamic changes of auroral events and improve the accuracy and precision of segmentation. This result not only verifies the effectiveness of the present invention in the task of auroral image sequence segmentation, but also provides an important reference basis for future auroral event analysis and research.

[0117]

Claims

1. An unsupervised event segmentation method for all-sky aurora videos based on bidirectional perception, characterized by: The steps include: Step 1: Input a set of all-sky aurora image sequences. After intensity adjustment and preprocessing (dark current removal, cropping, and rotation), the feature extractor extracts shallow sequence features of the aurora image sequence. A frame is randomly selected from the shallow sequence features as the target frame. The target frame and the T-1 frames before the target frame are selected to form a forward subsequence centered on the target frame. The target frame and the T-1 frames after the target frame are selected to form a backward subsequence centered on the target frame. To capture high-level feature representations of the aurora image sequence, contrastive learning is introduced for the forward and backward subsequences. Specifically, for the forward subsequence, the T-1 frames before the target frame are selected as positive samples; for the backward subsequence, the T-1 frames after the target frame are selected as positive samples. A multi-level negative sample selection strategy is also designed, including negative sample pairs with a large time span within the same video, cross-video negative sample pairs between different videos, and negative sample pairs embedded in memory during iterations. Finally, cosine similarity is used to measure similarity. Based on the similarity results, a contrast loss function is designed to bring positive samples with high similarity closer and negative samples with low similarity further away. Step 2: Based on the forward and backward subsequence features in step 1, a bidirectional feature reconstruction module is designed to reconstruct the target frame. The specific implementation is as follows: first, position encoding is added to the forward and backward subsequences after contrastive learning, and then a mask operation is performed on the target frames of the forward and backward subsequences. Finally, the modules are input into a reconstruction network composed of a multi-layer Transformer to obtain the forward target frame reconstructed from the forward subsequence and the backward target frame reconstructed from the backward subsequence. Minimizing the mean square error is used as the loss function, and the model training is supervised by calculating the pixel-level differences between the reconstructed forward and backward target frames and the original target frames to obtain a trained reconstruction model. Step 3, test the reconstruction model of step 2 to obtain the boundary probability curve; the specific implementation is: the reconstruction model sequentially scans the aurora image sequence from the beginning to the end, reconstructs the frame features frame by frame, obtains the reconstruction error of each frame, and forms an error curve; according to the continuity of the event, the error between the reconstructed forward target frame and the reconstructed backward target frame and the original target frame should be small, that is, the frame with a large reconstruction error indicates a high probability of belonging to the boundary frame, so the error curve is regarded as the boundary probability curve; the boundary probability curve is smoothed using a mean filter and a high-pass filter, and a boundary probability threshold is set. The frame that is greater than the boundary probability threshold and is a peak is regarded as a boundary frame.

2. The unsupervised event segmentation method for all-sky aurora video based on bidirectional perception according to claim 1 is characterized in that: The comparative study of the full-sky aurora image sequence described in step 1 is performed as follows: Step 1-1: Perform pre-processing operations on the full-sky aurora image sequence by intensity stretching, dark current subtraction, and cropping and rotation, and then perform shallow feature extraction to obtain shallow features X∈R B×C×H×W ; Step 1-2: Randomly select a frame from the sequence features to be segmented as the target frame, and select its previous T-1 frames to form a forward subsequence H1{x t-(T-1) ,...,x t-1 ,x t }, select the next T-1 frames to form a backward subsequence H2{x t ,x t+1 ,...,x t+(T-1) }; Step 1-3: For the forward subsequence, select the T-1 frames before the target frame as positive samples; for the backward subsequence, select the T-1 frames after the target frame as positive samples; there are three sources of negative samples, namely intra-video negative pairs (negative samples come from the same aurora image sequence but from different clips), inter-video negative pairs (negative samples are selected from any clips extracted from different aurora sequences) and memory negative pairs (negative samples come from frames embedded in the memory during the previous iteration); Steps 1-4: Use cosine similarity to measure similarity. Based on the similarity results, design a loss function to bring samples with high similarity closer and move away from samples with low similarity. The selected loss function is InfoNCE Loss in contrastive learning. Contrastive learning uses the similarity score of the positive sample pair as the numerator and the similarity scores of all positive and negative sample pairs as the denominator to calculate its negative log-likelihood loss, increasing the similarity score of the positive sample pair and decreasing the similarity score of the negative sample pair. See the formula as shown: in, Represents the contrast loss obtained by feature sequence H1 and feature sequence H2, represents the selection of the smallest value between the two for back propagation, q represents the query item, k i represents all sample keys, τ is the temperature parameter, k + is the positive sample key.

3. The unsupervised event segmentation method for all-sky aurora video based on bidirectional perception according to claim 1 is characterized in that: The target frame reconstruction described in step 2 includes the following steps: Step 2-1: Position encoding is performed on the two input subsequences, namely the forward subsequence and the backward subsequence. The position encoding is combined with the content encoding of the sequence to comprehensively represent the information of each position in the sequence; masking operations are performed on the two subsequences to obtain two masked subsequences; for the forward subsequence: the target frame is selected at the end of the forward subsequence, and the masking operation is performed on the tail frame to hide the information of the tail frame, so that the bidirectional feature reconstruction module cannot obtain the information of the tail frame; for the backward subsequence: the target frame is selected at the beginning of the backward subsequence, and the masking operation is performed on the first frame to hide the information of the first frame, so that the bidirectional feature reconstruction module cannot obtain the information of the first frame; the position encoding formula after masking is added as follows: H1′={h t-(T-1) ,...,h t-1 ,[MASK] t }+{pos t-(T-1) ,...,pos t-1 ,[MASK] t } H2′={[MASK] t ,h t+1 ,....,h t+(T-1) }+{[MASK] t ,pos t+1 ,...,pos t+(T-1) } Among them, H1' represents masking and adding position coding to the last frame of the forward subsequence, H2' represents masking and adding position coding to the first frame of the backward subsequence, MASK represents masking, and pos represents position coding; Step 2-2: The two subsequences masked in step 2-1 are input into a reconstruction network consisting of a multi-layer Transformer decoder. The forward reconstruction part is responsible for forward reconstruction of the masked tail frame. By learning the relationship between the previous frame and the tail frame, the bidirectional feature reconstruction module reconstructs the tail frame. The backward part is responsible for backward reconstruction of the first frame. By learning the relationship between the subsequent frames and the first frame, the bidirectional feature reconstruction module reconstructs the first frame. Step 2-3: By balancing the two reconstruction losses and the loss between the original features, the constraints on the reconstruction model are realized, and the mean square error loss is used to measure the reconstructed features, which is divided into bidirectional reconstruction loss and the original reconstruction loss The formula is: in represents the mean squared loss of the forward target frame and the backward target frame, Represents the minimum value of the mean square loss between the forward target frame and the original frame and the mean square loss between the backward target frame and the original frame, h t1 represents the forward reconstruction feature, h t2 represents the backward reconstruction feature, h t Represents the original frame features, s is the number of sequence frames, and V is the number of videos.

4. The unsupervised event segmentation method for all-sky aurora video based on bidirectional perception according to claim 1, characterized in that: The boundary probability curve described in step 3 is obtained by: Step 3-1: Replace the trained reconstruction model dataset with the test dataset, sequentially scan the aurora image sequence from the beginning to the end, reconstruct the frame features of the target frame, and obtain the reconstruction error of each frame to form an error curve. For the same event, due to the continuity of the event, the error between the reconstructed target frame and the original target frame is small. In other words, the frame with a large reconstruction error indicates a high probability of belonging to the boundary frame. Therefore, the error curve is regarded as the boundary probability curve. Step 3-2: Using a moving average filter, starting from the beginning of the signal sequence, select a data window of length N and calculate the average value of the data in the data window as the output at the current moment. Then, slide the data window backward by one data point and repeat the above calculation process until the entire signal sequence is processed. Use an FIR high-pass filter to remove the low-frequency noise boundary and retain the peak at the high-frequency event boundary. Step 3-3: Calculate the gradient of the reconstruction error trajectory to obtain the gradient trajectory, which reflects the rate of change of the reconstruction error; detect the relative maximum value on the gradient trajectory, and the relative extreme value points can be regarded as potential boundary locations; finally, perform threshold judgment on the detected relative extreme value points to distinguish which are true event boundaries and which are extreme values ​​caused by noise or local changes. Reasonable threshold screening can effectively improve the accuracy of boundary recognition.

Images