Method and device for identifying auroral substorm based on visual eye movement pattern and deep learning

Abstract

This invention discloses a method and apparatus for identifying auroral substorms based on visual eye-tracking patterns and deep learning, primarily addressing the problems of low accuracy and lack of basis for model design in existing auroral substorm identification methods. The implementation scheme includes: collecting eye-tracking data from multiple space physicists to generate expert eye-tracking gaze pattern maps for various types of auroral substorms; establishing an auroral substorm eye-tracking dataset; further designing a visual eye-tracking pattern prediction module to learn the experts' visual gaze patterns; and setting up a substorm sequence recognition module embedded with visual patterns, incorporating space physics knowledge and expert visual gaze patterns. Finally, a deep learning model suitable for auroral substorm sequence recognition is constructed and trained, and the trained model is used to identify auroral sequences. This invention not only enhances the representation of auroral substorm features but also improves the accuracy and rationality of auroral substorm image sequence recognition, and can be used for automated space physics event recognition.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision and space physics, and further relates to an automated method for identifying aurora substorms, which can be used for automated space physics event identification. Background Technology

[0002] Over the past few decades, machine learning methods have demonstrated their powerful feature extraction capabilities in space science. Initially, researchers used fuzzy c-means algorithms and shape-constrained sparse low-quality matrix factorization (MCF) from various types of aurora images to detect auroral substorms. With the rise of deep learning, convolutional neural networks (CNNs) have been applied to other tasks in space science, such as aurora image classification, aurora intensity modeling, space physics-based aurora event detection, and masked RCNN aurora image retrieval models based on saliency weights. Therefore, some researchers have begun using LSTM or C3D networks to extract image features from auroral substorm sequences, thereby achieving automated auroral substorm identification.

[0003] While current state-of-the-art LSTM and C3D-based auroral substorm identification models demonstrate good performance in automatically recognizing substorms, they fail to establish long-term inter-frame relationships, and their identified substorm sequences differ significantly from those identified by space physics experts. These models simply apply existing deep learning networks directly to auroral substorm identification without considering the specific characteristics of the images and the underlying space physics knowledge. Furthermore, while some Transformer-based image or video classification or recognition models perform very well on natural image tasks, they are unsuitable or incompatible for direct application to auroral substorm identification due to the significant differences between auroral substorm images and natural images. There is a lack of dedicated deep learning networks for auroral substorm sequence identification. When existing deep learning networks are directly used to identify auroral substorm sequences, their accuracy is limited.

[0004] Patent document CN201710937041.7 discloses "An Image Retrieval Method Based on Hierarchical Convolutional Neural Networks," which mainly addresses the problem of low accuracy in existing all-sky aurora image retrieval. Its implementation steps are: (1) using an adaptive fence method to determine local key points in the all-sky image; (2) extracting local SIFT features from the all-sky aurora image and constructing a visual dictionary; (3) pre-training and fine-tuning the convolutional neural network and constructing a polarization region pooling layer; (4) extracting CNN features from the all-sky aurora image region; (6) constructing an inverted index table and separately storing the global CNN features; (7) extracting the convolutional features of the query image and calculating its similarity to the database image, outputting the retrieval results. This method uses convolutional features to achieve matching between local key points, solving the problem of high false alarm rates in existing image retrieval methods. However, since it mainly targets all-sky aurora images and is an image classification method, it cannot identify substorm events based on ultraviolet aurora image sequences.

[0005] Patent document CN202311569468.8 discloses a method for retrieving auroral substorm images based on cross-modal spatial environmental parameters. The implementation steps are as follows: generating a substorm event training set composed of spatial environmental parameters and auroral images; constructing a deep supervised cross-modal retrieval network with a cross-attention mechanism; training the cross-modal retrieval network; and retrieving auroral substorm images using spatial environmental parameters. This method trains the network using a multi-head cross-attention mechanism and a deep supervised learning algorithm to avoid interference from redundant information in global features, extracting more accurate image features and parameter features; and retrieves auroral substorm images using spatial physical parameter information. However, since it mainly retrieves substorm images acquired by the SUSSI satellite, the designed deep learning network does not consider the physical characteristics of auroral images or the experience of space physics experts. Furthermore, the network requires corresponding spatial physical parameters as input, thus it cannot directly identify ultraviolet auroral image sequences. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the existing technology by proposing an automated aurora substorm identification method based on visual eye-tracking patterns and deep learning. This method integrates the physical characteristics of aurora images and the experience and knowledge of space physics experts into a deep learning network, thereby improving the accuracy of identifying substorm events based on ultraviolet aurora image sequences.

[0007] The technical solutions for achieving the above objectives include auroral substorm identification methods and auroral substorm identification devices:

[0008] I. The present invention provides a method for identifying aurora substorms based on visual eye-tracking patterns and deep learning, characterized in that it includes:

[0009] A space physics expert eye-tracking dataset was constructed, which includes the original ultraviolet aurora image sequence AS and some expert eye-tracking gaze pattern sequence ES.

[0010] Read each aurora image in the sequence AS I And its corresponding expert eye-tracking fixation pattern diagram ES I As the initial training set, each aurora image AS in the remaining sequence I As the initial test set;

[0011] A visual eye-tracking pattern prediction module, including feature extraction and prediction head, is constructed, and the sum of cross-entropy and linear correlation coefficient is used as the loss function L1 of this module;

[0012] A substorm sequence recognition module is constructed, which includes magnetic local time-geomagnetic latitude segmentation, feature embedding, feature encoding, and visual pattern embedding of the substorm recognition head. The cross-entropy function is used as the loss function L2 for this module.

[0013] The eye movement pattern prediction module is trained using the initial training set until the loss function converges, and the initial test set is input into the trained eye movement pattern prediction module to obtain its corresponding eye movement fixation pattern prediction map EP. I This leads to the aurora images in the initial test set being AS I And its corresponding eye-tracking fixation pattern prediction map EP I It is divided into a final training set and a final test set;

[0014] The visual pattern embedding subburst sequence recognition module is trained using the final training set until the loss function converges. The final test set is then input into the trained visual pattern embedding subburst sequence recognition module to obtain the sequence recognition results.

[0015] II. The present invention provides an aurora substorm recognition device based on visual eye-tracking patterns and deep learning, characterized in that it comprises:

[0016] The space physics expert eye-tracking data acquisition module 1 is used to collect eye-tracking gaze data of experts when viewing aurora substorm sequences and transmit the collected data to the expert eye-tracking dataset construction module 2;

[0017] Expert eye-tracking dataset construction module 2 is used to construct the initial training set and test set, and the visual eye-tracking pattern prediction module 3 is trained using the initial training set;

[0018] Visual eye movement pattern prediction module 3 is used to output an eye movement gaze pattern prediction map corresponding to the aurora image;

[0019] Substorm recognition dataset construction module 4 is used to construct the final training and testing sets; the visual pattern embedding substorm sequence recognition module 5 is trained and tested using the final training set.

[0020] The visual pattern-embedded substorm sequence recognition module 5 is used to output the recognition results of aurora substorm sequences.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] First, visual representations of expert experience can be used to help improve model performance.

[0023] Existing methods for identifying aurora substorms all directly utilize existing models or algorithms in the field of machine learning to extract features from aurora images. They all ignore the physical characteristics of aurora images and the experience and knowledge of space physics experts.

[0024] In this invention, eye-tracking patterns are considered a form of empirical knowledge. By collecting eye-tracking data from multiple space physicists, an aurora substorm eye-tracking dataset was established. By analyzing this eye-tracking data, expert eye-tracking gaze patterns for various types of aurora substorms were generated. These patterns are intuitive and easy to input into deep learning networks, thus helping to improve model performance.

[0025] Secondly, the model design is based on evidence and has a high recognition accuracy.

[0026] This invention incorporates expert subjective visual gaze patterns in the visual eye-tracking pattern prediction module and integrates scientific knowledge from the field of space physics into the sub-storm sequence recognition module embedded in the visual pattern. In other words, it designs a deep learning model suitable for aurora sub-storm sequence recognition by combining subjective and objective aspects. This not only provides enhanced representation of aurora sub-storm features but also effectively improves the accuracy of ultraviolet aurora sub-storm image sequence recognition and the rationality of the model design. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0028] Figure 2 This is a flowchart of the expert eye-tracking data acquisition sub-process in this invention;

[0029] Figure 3 This is a schematic diagram of the expert visual gaze pattern in this invention;

[0030] Figure 4 This is a structural diagram of the visual eye movement pattern prediction module in this invention;

[0031] Figure 5 This is a structural diagram of the substorm sequence recognition module with visual pattern embedding in this invention.

[0032] Figure 6 This is a schematic diagram of the magnetic local time-geomagnetic latitude block in this invention;

[0033] Figure 7This is a schematic diagram of the feature embedding submodule in this invention;

[0034] Figure 8 This is a structural diagram of the feature encoding submodule and the recognition head submodule in this invention;

[0035] Figure 9 This is a structural block diagram of the device of the present invention;

[0036] Figure 10 This is a subjective comparison chart of the eye movement gaze pattern prediction results generated by the present invention and existing methods. Detailed Implementation

[0037] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0038] One of the key technologies in this example is the setup consisting of two modules: a visual eye-tracking pattern prediction module and a visual pattern-embedded substorm sequence recognition module. The visual eye-tracking pattern prediction module aims to learn and mimic real expert eye-tracking patterns and is trained using eye movement data from experts observing aurora substorm sequences. The visual pattern-embedded substorm sequence recognition module is used to embed visual patterns and spatial physics knowledge into a deep learning network for substorm event recognition. A second key technology is the use of the transformer model, the optimal model for processing temporal data, as the network backbone, considering the temporal nature of substorm events. This improves the accuracy of the automatic aurora substorm recognition method while also increasing the rationality of the model design.

[0039] Example 1: A method for identifying sub-aurora storms based on visual eye-tracking patterns and deep learning

[0040] Reference Figure 1 The implementation steps of this example include the following:

[0041] Step 1: Collect eye-tracking data from space physics experts.

[0042] Reference Figure 2 The implementation of this step is as follows:

[0043] 1.1) The expert sits 50-100 cm away from the screen of the eye tracker, fixes the subject's head, and uses the 9-point calibration method to calibrate the eyeballs; according to the introductory text displayed on the screen, press the button to start playing the aurora image sequence;

[0044] 1.2) After the expert has viewed the current sequence, record the judgment result in the experimental record book and end the experiment; if the expert cannot give a judgment result for the sequence after this viewing, return to step 1.1) to view it again and record the judgment result, and end the experiment.

[0045] 1.3) Export the expert's gaze eye movement information document, which contains the expert's gaze point location and the duration of each gaze point;

[0046] In this example, a total of 7 experts participated in the collection of eye-tracking data. Each expert repeated steps 1.1)-1.3) until the collection of eye-tracking data from all experts was completed.

[0047] Step 2: Convert the expert eye-tracking data into an expert eye-tracking fixation pattern diagram.

[0048] To utilize the expert's visual eye-tracking information, the eye-tracking data needs to be converted from two-dimensional points into a visual representation, that is, mapping the acquired coordinates of the expert's gaze position to the aurora substorm image AS. I On a matrix of uniform size, an expert eye-tracking gaze pattern map (ES) is then generated. I The conversion from two-dimensional points to visual representation is completed, and its implementation includes the following:

[0049] 2.1) First, generate an all-zero matrix with the same size as the aurora image.

[0050] 2.2) Mark the location of the expert eye-tracking fixation point on the all-zero matrix, set the pixel value of the expert eye-tracking fixation location to 255, and then assign the value of 255 to all positions within a circle with a radius of 10 centered on this location.

[0051] 2.3) Repeat 2.2) until all expert gaze points on the aurora image are marked in the all-zero matrix, and obtain the eye-tracking gaze pattern diagram of the aurora image;

[0052] 2.4) Repeat steps 2.1)-2.3) to generate the corresponding expert eye-tracking fixation pattern map for all aurora images in each substorm sequence, such as... Figure 3 As shown. Among them Figure 3 The first row is a sequence of substorm images, and the second row is its corresponding expert eye-tracking gaze pattern (ES). I .

[0053] Step 3: Construct a visual eye-tracking pattern prediction module.

[0054] This example demonstrates how to build a visual eye-tracking pattern prediction module using a simple network built on the Transformer architecture to generate an eye-tracking fixation pattern prediction map (EP). I .

[0055] Reference Figure 4 The visual eye-tracking pattern prediction module includes a feature extraction submodule and a prediction head submodule. The structure of each submodule is as follows:

[0056] The feature extraction submodule, which consists of a ResNet50 network and a ViT visual converter network connected in series, is used to extract image features, namely the aurora image AS. I Primary image features I are extracted using the ResNet50 model. R The primary image feature I R High-level image features extracted using the Vit network, a visual converter. V The size is 4096*512;

[0057] The prediction head submodule consists of a latitudinal transformation roll, a linear transformation roll, a normalization roll, a two-dimensional convolutional roll, and an upsampling roll, concatenated together. It is used to generate an eye-tracking gaze pattern prediction map, that is, to extract high-level image features from the ViT network. V The input to this module, after each volume, results in the following feature sizes: 512*4096, 512*64*64, 512*64*64, 1*64*64, and 1*241*241. The last two dimensions of the final volume are taken as the final generated eye-tracking fixation pattern prediction map EP. I .

[0058] Step 4: Set the loss function L1 for the visual eye-tracking pattern prediction module.

[0059] Calculate the correlation coefficient: CC(EP) I ES I ) = cov(EP I ES I ) / δ ES *δ EP Where cov() represents the covariance function, δ ES and δ EP These represent expert eye-tracking fixation patterns (ES). I Eye-tracking fixation pattern prediction map EP I Standard deviation;

[0060] Calculate cross-entropy: Where i represents the expert eye-tracking fixation pattern diagram ES I Eye-tracking fixation pattern prediction map EP I Pixels in;

[0061] The loss function L1 for the visual eye-tracking pattern prediction module is defined based on the calculated correlation coefficient and cross-entropy:

[0062] L1 = CC(EP) I ES I )+H(EP I ES I ).

[0063] Step 5: Train the eye-tracking pattern prediction module using the initial training set.

[0064] To facilitate the network's better differentiation and learning of eye movement patterns at different stages of substorm image sequences, this module does not consider the temporal nature of the sequence, and its training steps include:

[0065] 5.1) Read the aurora images AS from the initial training set respectively. I And its corresponding expert eye-tracking fixation pattern diagram ES I ;

[0066] 5.2) The expert eye-tracking fixation pattern diagram ES I Convert to grayscale image and use the aurora images AS from the initial training set. I Corresponding expert eye-tracking fixation pattern diagram ES I Convert from BGR format to RGB format and then to tensor;

[0067] 5.3) Set the size of the aurora sequence images in the initial dataset to 241*241*3, the size of the corresponding expert eye-tracking gaze pattern image to 241*241, the batch size to 1, the optimizer to SGD, the number of training epochs to 400, the initial learning rate to 0.003, and the momentum to 0.9.

[0068] 5.4) Input a batch of aurora images into the eye-tracking pattern prediction module to obtain the eye-tracking gaze pattern prediction map for that batch; calculate the loss value using the eye-tracking gaze pattern prediction map and the expert eye-tracking gaze pattern map according to the loss function L1, and backpropagate the loss value using the SGD optimizer to update the network parameters;

[0069] 5.5) Repeat step 5.4), reducing the learning rate by a factor of 10 every 20 iterations, until the maximum number of training rounds of 400 is reached or the loss function stops decreasing within 10 iterations, and the trained eye movement pattern prediction module is obtained.

[0070] Step 6: Generate an eye-tracking pattern prediction map using the initial test set, and divide the final training set and the final test set.

[0071] 6.1) Read the aurora images AS from the initial test set respectively. I Then convert it from BGR format to RGB format and then to tensor;

[0072] 6.2) Add all aurora images in the test set to AS I The data is sequentially input into the trained eye movement pattern prediction module to obtain the corresponding eye movement fixation pattern prediction map (EP). I ;

[0073] 6.3) Convert the above aurora image AS I Eye-tracking fixation pattern prediction map EP IArrange the sequence according to its timestamp, and then divide the eye-tracking gaze pattern prediction map sequence and all aurora image sequences into the final training set and the final test set in a 2:1 ratio.

[0074] Step 7: Construct a visual pattern-embedded substorm sequence recognition module to identify whether the input aurora sequence is a substorm sequence.

[0075] Physicists typically convert auroral substorm images into magnetolocal time (MLT) and magnetomagnetic latitude (MLAT) coordinate systems for analysis. Within these coordinate systems, the dynamic changes of bright spots on the auroral ellipse can be observed using a unified standard. Therefore, segmentation and encoding require processing along both directions separately.

[0076] Reference Figure 5 The visual pattern embedding substorm sequence recognition module constructed in this example includes: a magnetic local time-geomagnetic latitude segmentation submodule, a feature embedding submodule, a feature encoding submodule, and a substorm recognition head submodule.

[0077] The magnetic local time geomagnetic latitude segmentation submodule is a novel image segmentation method, such as... Figure 6 As shown, it consists of parallel magnetic local time (MLT) blocks and geomagnetic latitude (MLAT) blocks. The magnetic local time blocks divide the image into four main regions according to magnetic local time, representing noon, morning, dusk, and night, resulting in magnetic local time block E. MLT ;

[0078] The geomagnetic latitude segmentation is based on the fact that auroral substorms typically occur between 50 and 90 degrees. When segmenting the geomagnetic latitude, the image is divided into concentric circles, meaning a 10-degree segment is taken from 50 to 90 degrees, resulting in different geomagnetic latitude blocks, including geomagnetic latitude blocks E within the 80-90 degree range. MLAT89 Geomagnetic latitude block E in the range of 70-80 degrees MLAT78 Geomagnetic latitude block E in the range of 60-70 degrees MLAT67 Geomagnetic latitude block E within the range of 50-60 degrees MLAT56 ;

[0079] The feature embedding submodule consists of a flattened volume, a spliced ​​volume, two normalized volumes, a fully connected volume, and a position code concatenated and then concatenated with another position code in parallel. Figure 7 As shown.

[0080] The concatenated portion is used to transform the geomagnetic latitude block to the same size as the local magnetic time block and embed its features. Assuming the dimensions of each vector are d1, d2, d3, and d4, these four image block vectors are concatenated using a concat operation to obtain an image block vector with a dimension of 4*(d1+d2+d3+d4). After passing through two normalized convolutions and one fully connected convolution, a feature of the aurora image after geomagnetic latitude embedding is obtained, with a dimension of 4*1024. Adding positional encoding to this feature yields a geomagnetic latitude feature E of size 5*1024. MLAT .

[0081] The parallel section is used to perform position encoding on the magnetic local time block, which also yields a magnetic local time feature E of size 5*1024. MLT ;

[0082] The feature encoding submodule consists of two sets of token allocations, a Transformer encoder, a feature concatenation volume, a max-pooling volume, and a Transformer encoder connected in parallel and then concatenated with a feature concatenation volume. Figure 8 As shown.

[0083] This module is used to encode the embedding features of the aurora substorm sequence and the embedding features of the gaze pattern prediction map sequence to obtain the fused feature T. AV Its specific implementation includes the following:

[0084] For each aurora image sequence and its corresponding eye-tracking fixation pattern prediction map sequence, after performing magnetolocal time geomagnetic latitude segmentation and feature embedding, the magnetolocal time geomagnetic latitude embedding features are obtained as follows: Magnetolocal time embedding features of aurora substorm sequences. Geomagnetic latitude embedding features of auroral substorm sequences Magnetic local time embedding features of gaze prediction map sequence Geomagnetic latitude embedding features of gaze prediction map sequence The size of all features is represented as (b*f*p*d), where b represents the batch size; f is the number of frames or the size of a sub-block; p represents the number of blocks + 1, where 1 represents positional encoding; and d is the embedding dimension.

[0085] Magnetic local time embedding features of auroral substorm sequences Geomagnetic latitude embedding features of auroral substorm sequences Magnetic local time embedding features of gaze prediction map sequence Geomagnetic latitude embedding features of gaze prediction map sequence Assign the corresponding tokens respectively, and represent them as follows: and

[0086] The tokens assigned to the embedded features of the aurora image sequence are encoded using a Transformer encoder to obtain the aurora substorm sequence features T. AS (10*5*2048); that is, the local time embedding feature of the magnetic field. Allocated and geomagnetic latitude embedding features Allocated Attention calculations are performed, and then the images are stitched together frame by frame. and Obtain the token feature T of the aurora substorm sequence AS (10*5*2048);

[0087] The tokens assigned to the embedded features of the gaze pattern prediction image sequence are encoded using a Transformer encoder to obtain the gaze prediction map sequence features T. VS (10*5*2048), that is, first embedding the magnetic local time features separately. Allocated and geomagnetic latitude embedding features Allocated Attention calculations are performed using a 10*5*1024 matrix; then frame-by-frame stitching is performed. and Obtain the token features T of the gaze prediction map sequence VS (10*5*2048);

[0088] The aurora substorm sequence characteristics T AS (10*5*2048) and gaze prediction map sequence features T VS (10*5*2048) are fused, that is, a max-pooling convolution and a Transformer encoder are used to reduce its feature dimension, so that T AS The dimension becomes (10*3*1024), T VS The dimensions become (10*3*128); then T AS (10*3*1024) and T VS (10*3*128) Perform a concatenation operation on the second dimension to obtain the fused feature T. AV (10*3*1152).

[0089] The substorm identification head module consists of an average pooling volume, a Transformer encoder, and a multi-volume perceptron connected in series, as follows: Figure 8As shown, it is used to fuse feature T AV Encode and classify the fused features T AV (10*3*1152) Input this recognition head to get the final recognition result; the feature size changes after each roll as follows: 10*3*1152, 10*1*576, 10*576, 10*2.

[0090] Step 8: Based on the substorm sequence recognition results and the true sequence labels, set the loss function L2 of the visual pattern embedded substorm sequence recognition module.

[0091]

[0092] Among them, Q j P represents the true label of the sequence. j This represents the substorm sequence identification result, where j represents the sequence label Q. j And sub-storm sequence identification results P j The number of sequences.

[0093] Step 9: Process the aurora image sequences and eye-tracking fixation pattern prediction map sequences from the final training set and the final test set into sequences of equal length S. p :

[0094]

[0095] Where p and k represent the sequence lengths before and after processing, respectively, and L is the set sequence length, which is set to 10 in this example; finally, each training sample includes an aurora substorm image sequence and the corresponding eye-tracking gaze pattern prediction map arranged in order.

[0096] Step 10: Train the visual pattern embedding subburst sequence recognition module using the final training set.

[0097] 10.1) Read all images in the eye-tracking gaze pattern prediction map sequence in the final training set, convert them to grayscale images and then to tensors; read all images in the aurora sequence in the final training set, convert them from BGR format to RGB format and then to tensors.

[0098] 10.2) Set the aurora sequence size in the initial dataset to 10*241*241, the batch size to 10, the optimizer to Adam, the number of training epochs to 100, the initial learning rate to 0.001, and the momentum to 0.9.

[0099] 10.3) Input a batch of aurora sequences and eye-tracking fixation pattern prediction map sequences into the visual pattern embedding substorm sequence recognition module to obtain the sequence recognition result of the batch; use the recognition result and the real label to calculate the loss value according to the loss function L2, and use the Adam optimizer to backpropagate the loss value to update the network parameters.

[0100] 10.4) Repeat step 10.3), and reduce the learning rate by a factor of 10 every 10 iterations until the maximum number of training rounds of 100 is reached or the loss function no longer decreases within 10 iterations. Training stops, and the trained visual pattern embedding sub-burst sequence recognition module is obtained.

[0101] Step 11: Obtain the aurora substorm identification results using the final test set.

[0102] Read all images from the aurora sequence in the final test set, convert them from BGR format to RGB format, and then convert them into tensors;

[0103] Then, it is input into the trained visual pattern embedding substorm sequence recognition module to obtain the recognition result of whether the sequence is a substorm sequence.

[0104] Example 2: Aurora Substorm Recognition Device Based on Visual Eye-Motion Patterns and Deep Learning

[0105] Reference Figure 9 The aurora substorm identification device in this example includes: a space physics expert eye-tracking data acquisition module 1, an expert eye-tracking dataset construction module 2, a visual eye-tracking pattern prediction module 3, a substorm identification dataset construction module 4, and a visual pattern embedded substorm sequence identification module 5.

[0106] The space physics expert eye-tracking data acquisition module 1 is used to collect eye-tracking fixation data when experts view aurora substorm sequences, and the collected data is transmitted to the expert eye-tracking dataset construction module 2.

[0107] The expert eye-tracking dataset construction module 2 includes an expert eye-tracking gaze pattern generation submodule 21 and a data partitioning submodule 22. The eye-tracking gaze pattern generation submodule 21 generates expert eye-tracking gaze pattern maps using the collected expert eye-tracking gaze data. The data partitioning submodule 22 divides the aurora image and its corresponding expert eye-tracking gaze pattern map into an initial training set and a test set. These initial training set and test set are input into the visual eye-tracking pattern prediction module 3 for module training.

[0108] The visual eye movement pattern prediction module 3 includes: a feature extraction submodule 31 and a prediction head submodule 32; wherein, the feature extraction submodule 31 is used to extract aurora image features, and the extracted aurora image features are input into the prediction head submodule 32 to generate an eye movement gaze pattern prediction map; this module and the eye movement gaze pattern prediction map sequence output from the aurora image sequence in the initial test are used to input into the substorm recognition dataset construction module 4;

[0109] The substorm recognition dataset construction module 4 is used to construct the final training set and test set, which includes aurora image sequences, eye-tracking gaze pattern prediction map sequences, and sequence real recognition labels, for training and testing of the visual pattern embedding substorm sequence recognition module 5.

[0110] The visual pattern embedding substorm sequence recognition module 5 includes: a magnetic local time-geomagnetic latitude segmentation submodule 51, a feature embedding submodule 52, a feature encoding submodule 53, and a substorm recognition head submodule 54; wherein, the magnetic local time-geomagnetic latitude segmentation submodule 51 is used to segment the aurora image sequence and the eye-tracking gaze pattern prediction map sequence according to their spatial physical attributes; the feature embedding submodule 52 is used to embed features into the magnetic local time-geomagnetic latitude blocks of the aurora image and the magnetic local time-geomagnetic latitude blocks of the eye-tracking gaze pattern prediction map; the embedded features are used to input the feature encoding submodule 53 for feature encoding and fusion; the fused features are used to input the substorm recognition head submodule 54 to output the recognition result of the aurora substorm sequence.

[0111] The effects of the present invention will be further described in detail below with reference to simulation experiments.

[0112] 1. Simulation Experiment Conditions

[0113] The computer used has an Intel(R) Core(TM) i5-6600 CPU @ 3.30GHz, 16GB of RAM, and two NVIDIA RTX 3090 GPUs with 24GB of video memory each.

[0114] The operating system is 64-bit Ubuntu 18.04, and the algorithm simulation uses Python, specifically version 2.1.0 of the deep learning framework PyTorch.

[0115] The evaluation metrics are Accuracy, Precision, Recall, and F-score, which are commonly used in behavior recognition.

[0116] Accuracy represents the proportion of all correctly predicted categories out of the total number of categories, and is calculated as follows:

[0117]

[0118] In the formula, TP is the number of positive samples predicted as positive samples, FN is the number of positive samples predicted as negative samples, FP is the number of negative samples predicted as positive samples, and TN is the number of negative samples predicted as negative samples.

[0119] Precision represents the proportion of positive samples in a set of predicted pairs, and it is calculated as follows:

[0120]

[0121] Recall represents the proportion of positive samples in a predicted pair among all real positive samples, and it is calculated as follows:

[0122]

[0123] The F-score is a unified metric for measuring a model's recall and precision, and it is calculated as follows:

[0124]

[0125] When β is 1, the F-score is the commonly used F1.

[0126] 2. Simulation Experiment Content and Results

[0127] Experiment 1: Under the above simulation experimental conditions, the eye-tracking gaze prediction methods of this invention and existing methods were trained on the initial training set collected in step 1, and tested using 58 aurora image sequences from the initial test set to obtain subjective results of eye-tracking gaze pattern prediction, such as... Figure 10 As shown.

[0128] from Figure 10 As can be seen from the above, compared with other methods, the eye movement pattern prediction module of the present invention has a smaller fixation point prediction area, a more accurate fixation point position, and is closer to the eye movement fixation pattern diagram of experts.

[0129] Experiment 2: Under the above simulation experimental conditions, the substorm sequence recognition module embedded in the visual pattern of this invention and six existing substorm image sequence recognition methods were used for training and testing, respectively. The sequence recognition results were obtained, and the results were evaluated using common classification evaluation metrics. The results are shown in Table 1.

[0130] Table 1. Comparison of performance indicators between the present invention and six existing methods for substorm sequence identification.

[0131] method Accuracy↑ Precision↑ Recall↑ F1 score↑ Vit-3d 0.9134 0.8846 0.92 0.9020 Video-Swin-tiny 0.8304 0.8 0.81 0.8050 Video-Swin-small 0.6391 0.5702 0.75 0.6479 Video-FocalNet 0.9091 0.8692 0.93 0.8986 DualFormer-tiny 0.8609 0.8384 0.83 0.8342 Yang's - 0.4928 0.9198 0.6417 EMSF-R2Plus1D 0.8826 0.8462 0.88 0.8627 EMSF-C3D 0.9087 0.8911 0.9 0.8955 This invention 0.9264 0.9029 0.93 0.9163

[0132] As can be seen from Table 1, the visual pattern embedding substorm sequence recognition module of the present invention outperforms existing methods in the above indicators, indicating that the present invention can better identify substorm sequences.

[0133] The simulation results above demonstrate that the present invention has higher accuracy and robustness in the identification of space physical events.

[0134] The sources of the above six comparison methods are as follows:

[0135] Vit-3d is a method from a 2020 paper published on arXiv titled "An image is worth 16x16 words: Transformers for image recognition at scale".

[0136] Video-Swin-tiny and Video-Swin-small are methods from the paper titled Video SwinTransformer published on the arXiv website;

[0137] Video-FocalNet is a method presented in a paper titled "Video-FocalNets: Spatio-Temporal Focal Modulation for Video Action Recognition" published in the Proceedings of the IEEE International Conference on Computer Vision in 2023.

[0138] DualFormer-tiny is a method presented in a paper titled "DualFormer: Local-Global Stratified Transformer for Efficient Video Recognition" published in the Proceedings of the IEEE International Conference on Computer Vision in 2023.

[0139] Yang's method is the one published in 2014 in the IEEE Transactions on Image Processing paper "Improving level set method for fast auroral oval segmentation".

[0140] EMSF-R2Plus1D and EMSF-C3D are methods presented in a paper titled "Progressively Coding the Human Visual Information for Auroral Substorm Recognition" published at the International Conference on Frontiers of Electronics, Information and Computation Technologies in 2023.

[0141] The above description is merely a specific example of the present invention and does not constitute any limitation on the present invention. It demonstrates that those skilled in the art, after understanding the content and principles of the present invention, may make various modifications and changes in form and detail without departing from the principles and structure of the present invention. For example, the eye-tracking data processing method of the present invention, besides visualizing the expert gaze point as an expert eye-tracking gaze pattern diagram in this example, can be adjusted for different types of data; the feature extractors involved in the present invention, except for the Resent50-ViT used in this example, can all be replaced by other feature extraction modules. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0142] It should be noted that the step numbers in this specification and claims are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.

Claims

1. An automated method for identifying sub-aurora storms based on visual eye-tracking patterns and deep learning, characterized in that, include: A space physics expert eye-tracking dataset was constructed, which contains raw ultraviolet aurora image sequences. And some expert eye-tracking fixation pattern sequences ; Read each aurora image in the sequence And its corresponding expert eye-tracking fixation pattern diagram Each aurora image in the remaining sequence serves as the initial training set. As the initial test set; A visual eye-tracking pattern prediction module is constructed, which includes feature extraction and prediction head, and the sum of cross-entropy and linear correlation coefficient is used as the loss function of this module. ; A substorm sequence recognition module is constructed, which includes magnetic local time-geomagnetic latitude segmentation, feature embedding, feature encoding, and visual pattern embedding of a substorm recognition head. The cross-entropy function is used as the loss function for this module. ; The eye movement pattern prediction module is trained using the initial training set until the loss function converges, and the initial test set is then input into the trained eye movement pattern prediction module to obtain its corresponding eye gaze pattern prediction map. This will allow the aurora images in the initial test set to be used for further testing. And its corresponding eye-tracking fixation pattern prediction map It is divided into a final training set and a final test set; The visual pattern embedding subburst sequence recognition module is trained using the final training set until the loss function converges, and the final test set is input into the trained visual pattern embedding subburst sequence recognition module to obtain the sequence recognition result. The visual pattern-embedded substorm sequence recognition module includes: a magnetic local time-geomagnetic latitude segmentation submodule, a feature embedding submodule, a feature encoding submodule, and a substorm recognition head submodule. This magnetic local time-geomagnetic latitude segmentation submodule consists of magnetic local time (MLT) segments and geomagnetic latitude (MLAT) segments connected in parallel. The magnetic local time segment is used to divide the image into four regions according to magnetic local time, representing noon, morning, dusk, and night, respectively, to obtain magnetic local time blocks. Geomagnetic latitude segmentation is used to divide the image into concentric circles, taking a block every 10 degrees from 50 degrees to 90 degrees to obtain different geomagnetic latitude blocks. , , , ; This feature embedding submodule consists of a flattening layer, a stitching layer, two normalization layers, a fully connected layer, and a location code concatenated and then paralleled; the concatenated part is used to convert the dimensions of the geomagnetic latitude block to those of the magnetic local time block. The same size is used for feature embedding; the parallel part is used for feature embedding of the magnetic local time block; the magnetic local time features are obtained respectively. Geomagnetic latitude characteristics ; This feature encoding submodule consists of two sets of token allocations, a Transformer encoder, a feature concatenation layer, a max pooling layer, and a Transformer encoder connected in parallel and then concatenated with another feature concatenation layer. It is used to encode the embedded features of the aurora substorm sequence and the embedded features of the gaze pattern prediction map sequence to obtain the fused features. ; This sub-burst recognition head module consists of an average pooling layer, a Transformer encoder, and a multilayer perceptron connected in series, used to fuse features. Encode and classify the sequences to obtain the identification results.

2. The method according to claim 1, characterized in that, The construction of the space physics expert eye-tracking dataset includes: 2a) Collect eye-tracking data from experts: 2a1) The expert sits 50-100 cm away from the screen of the eye tracker, fixes the subject's head, and uses the 9-point calibration method to calibrate the eyeballs; 2a2) Press the button to start playing the aurora image sequence according to the intro displayed on the screen; 2a3) After the expert has finished viewing the current sequence, record the judgment result in the experimental record book and end the experiment; if the expert cannot give a judgment result for the sequence after this viewing, repeat steps 2a1)-2a2) to view it again and record the judgment result, and end the experiment. 2a4) Export the expert's gaze eye movement information document, which contains the expert's gaze point location and the duration of each gaze point; 2b) Convert eye-tracking data into expert eye-tracking fixation pattern diagrams : 2b1) Generate an all-zero matrix with the same size as the aurora image, and set the pixel value of the expert eye-tracking fixation position to 255 in the all-zero matrix; 2b2) Using each fixation point as the center, assign a value of 255 to all pixels within a circle with a radius of 10 to obtain the expert eye-tracking fixation pattern diagram of the aurora image. 2b3) For all images in each aurora sequence, repeat steps 2b1) to 2b2) to obtain all aurora images. Generate the corresponding expert eye-tracking gaze pattern diagram. .

3. The method according to claim 1, characterized in that, The visual eye-tracking pattern prediction module includes: a feature extraction submodule and a prediction head submodule. This feature extraction submodule, consisting of a ResNet50 network cascaded with a ViT visual transformer network, is used to extract image features, specifically aurora images. ResNet50 model extracts primary image features This primary image feature High-level image features extracted using the Vit network, a visual converter. ; This prediction head submodule, It consists of a dimensionality transformation layer, a linear transformation layer, a normalization layer, a two-dimensional convolutional layer, and an upsampling layer connected in series, and is used to process high-level features. The input prediction head submodule produces the final eye-tracking gaze pattern prediction map. .

4. The method according to claim 1, characterized in that, The visual eye-tracking pattern prediction module loss function It is expressed as follows: ， in The correlation coefficient is... Represents the covariance function. and These represent expert eye-tracking fixation patterns. Eye-tracking fixation pattern prediction map Standard deviation; For cross-entropy, Diagram showing expert eye-tracking fixation patterns Eye movement fixation pattern prediction map The pixels in the image.

5. The method according to claim 1, characterized in that, The loss function of the substorm sequence recognition module embedded in the visual pattern , means as follows: ， in, Represents sequence label Results of sub-storm sequence identification The number of sequences.

6. The method according to claim 1, characterized in that, The step of training the eye-tracking pattern prediction module using an initial training set includes: 7a) Expert eye-tracking fixation pattern diagram Convert to grayscale; convert aurora images from the initial training set. Corresponding expert eye-tracking fixation pattern diagram Convert from BGR format to RGB format and then to tensor; 7b) Set the aurora sequence image size in the initial dataset to 241*241, the batch size to 1, the optimizer to SGD, the number of training epochs to 400, the initial learning rate to 0.003, and the momentum to 0.

9. 7c) Input a batch of aurora images into the eye-tracking pattern prediction module to obtain the eye-tracking gaze pattern prediction map for that batch; based on the loss function... The loss value is calculated using the eye-tracking gaze pattern prediction map and the expert eye-tracking gaze pattern map, and the loss value is backpropagated using the SGD optimizer to update the network parameters. 7d) Repeat step 7c), reducing the learning rate by a factor of 10 every 20 iterations, until the maximum number of training rounds of 400 is reached or the loss function stops decreasing within 10 iterations, and the trained eye-tracking pattern prediction module is obtained.

7. The method according to claim 1, characterized in that, The aurora images in the initial test set And its corresponding eye-tracking fixation pattern prediction map The process is divided into a final training set and a final test set. The aurora image sequence from the initial test set is input into the trained eye movement pattern prediction module to obtain the corresponding eye movement gaze pattern prediction map sequence. Then, the eye movement gaze pattern prediction map sequence and all aurora image sequences are divided into a final training set and a final test set in a 2:1 ratio.

8. The method according to claim 1, characterized in that, The substorm sequence recognition module, which is trained using the final training set to embed visual patterns, includes: 9a) Process the aurora image sequences and eye-tracking gaze pattern prediction map sequences in the final training set and the final test set into sequences of equal length. : ， in, and These represent the sequence lengths before and after processing, respectively. This is the set sequence length; 9b) The processed eye-tracking fixation pattern prediction sequence Convert to grayscale images; read all images from the aurora sequence and eye-tracking gaze pattern prediction map sequence in the final training set, convert them from BGR format to RGB format, and then convert them to tensors; 9c) Set the aurora sequence size in the initial dataset to 10*241*241, the batch size to 10, the optimizer to Adam, the number of training epochs to 100, the initial learning rate to 0.001, and the momentum to 0.

9. 9d) A batch of aurora sequences and eye-tracking fixation pattern prediction sequences are simultaneously input into a visual pattern embedding substorm sequence recognition module to obtain the sequence recognition result for that batch; this recognition result is then compared with the true labels according to the loss function. Calculate its loss value, and backpropagate the loss value using the Adam optimizer to update the network parameters; 9e) Repeat step 9d), reducing the learning rate by a factor of 10 every 10 iterations until the maximum number of training rounds of 100 is reached or the loss function no longer decreases within 10 iterations. Training then stops, and the trained visual pattern embedding subburst sequence recognition module is obtained.

9. A device for recognizing aurora substorms based on visual eye-tracking patterns and deep learning, characterized in that, include: The space physics expert eye-tracking data acquisition module 1 is used to collect eye-tracking gaze data of experts when viewing aurora substorm sequences and transmit the collected data to the expert eye-tracking dataset construction module 2; Expert eye-tracking dataset construction module 2 is used to construct the initial training set and test set, and the visual eye-tracking pattern prediction module 3 is trained using the initial training set; Visual eye movement pattern prediction module 3 is used to output an eye movement gaze pattern prediction map corresponding to the aurora image; Substorm recognition dataset construction module 4 is used to construct the final training and testing sets; the visual pattern embedding substorm sequence recognition module 5 is trained and tested using the final training set. The visual pattern-embedded substorm sequence recognition module 5 is used to output the recognition results of aurora substorm sequences; The visual pattern-embedded substorm sequence recognition module includes: a magnetic local time-geomagnetic latitude segmentation submodule, a feature embedding submodule, a feature encoding submodule, and a substorm recognition head submodule. This magnetic local time-geomagnetic latitude segmentation submodule consists of magnetic local time (MLT) segments and geomagnetic latitude (MLAT) segments connected in parallel. The magnetic local time segment is used to divide the image into four regions according to magnetic local time, representing noon, morning, dusk, and night, respectively, to obtain magnetic local time blocks. Geomagnetic latitude segmentation is used to divide the image into concentric circles, taking a block every 10 degrees from 50 degrees to 90 degrees to obtain different geomagnetic latitude blocks. , , , ; This feature embedding submodule consists of a flattening layer, a stitching layer, two normalization layers, a fully connected layer, and a location code concatenated and then paralleled; the concatenated part is used to convert the dimensions of the geomagnetic latitude block to those of the magnetic local time block. The same size is used for feature embedding; the parallel part is used for feature embedding of the magnetic local time block; the magnetic local time features are obtained respectively. Geomagnetic latitude characteristics ; This feature encoding submodule consists of two sets of token allocations, a Transformer encoder, a feature concatenation layer, a max pooling layer, and a Transformer encoder connected in parallel and then concatenated with another feature concatenation layer. It is used to encode the embedded features of the aurora substorm sequence and the embedded features of the gaze pattern prediction map sequence to obtain the fused features. ; This sub-burst recognition head module consists of an average pooling layer, a Transformer encoder, and a multilayer perceptron connected in series, used to fuse features. Encode and classify the sequences to obtain the identification results.

10. The apparatus according to claim 9, characterized in that: The expert eye-tracking dataset construction module 2 includes an expert eye-tracking gaze pattern generation submodule 21 and a data partitioning submodule 22; wherein, the eye-tracking gaze pattern generation submodule 21 is used to generate expert eye-tracking gaze pattern maps using the collected expert eye-tracking gaze data; the data partitioning submodule 22 is used to divide the aurora image and its corresponding expert eye-tracking gaze pattern map into an initial training set and a test set; The visual eye movement pattern prediction module 3 includes a feature extraction submodule 31 and a prediction head submodule 32; wherein, the feature extraction submodule 31 is used to extract aurora image features, and the extracted aurora image features are input into the prediction head submodule 32 to generate an eye movement gaze pattern prediction map. The visual pattern embedding substorm sequence recognition module 5 includes a magnetic local time-geomagnetic latitude segmentation submodule 51, a feature embedding submodule 52, a feature encoding submodule 53, and a substorm recognition head submodule 54. The magnetic local time-geomagnetic latitude segmentation submodule 51 is used to segment the aurora image sequence and the eye-tracking fixation pattern prediction map sequence according to their spatial physical attributes. The feature embedding submodule 52 is used to embed features into the magnetic local time-geomagnetic latitude blocks of the aurora image and the magnetic local time-geomagnetic latitude blocks of the eye-tracking fixation pattern prediction map. The feature encoding submodule 53 is used to encode and fuse the embedded features. The feature substorm recognition head module 54 is used to discriminate the fused features and output the recognition result of the aurora substorm sequence.

Images