Aurora substorm identification method and device based on visual eye movement mode and deep learning

Abstract

The invention discloses an aurora and substorm identification method and device based on a visual eye movement mode and deep learning, and mainly solves the problems that an existing aurora and substorm identification method is not high in accuracy and has no basis for model design. According to the implementation scheme, the method comprises the following steps: collecting eye movement data of a plurality of spatial physicians to generate expert eye movement fixation mode patterns of various types of aurora substorms; establishing an aurora sub-storm eye movement data set; a visual eye movement mode prediction module is further designed to learn a visual gazing mode of an expert, a visual mode embedded sub-storm sequence identification module is arranged, spatial physical knowledge and the visual gazing mode of the expert are integrated in the module, and finally a deep learning model suitable for aurora sub-storm sequence identification is constructed and trained. And identifying the aurora sequence by using the trained model. The method not only enhances the representation of the aurora and substorm features, but also improves the recognition accuracy of the aurora and substorm image sequence and the reasonability of model design, and can be used for automatic spatial physical event recognition.

Description

Technical Field

[0001] The present invention belongs to the interdisciplinary field of computer vision and space physics science, and further relates to an automated auroral substorm recognition method, which can be used for automated space physics event recognition. Background Art

[0002] In the past few decades, machine learning methods have shown their powerful feature extraction capabilities in the field of space science. Initially, researchers used the fuzzy c-means algorithm and shape-constrained sparse low-quality matrix decomposition in traditional machine learning methods to extract various features from various types of aurora images to detect auroral substorms. With the rise of deep learning, convolutional neural networks (CNNs) have been applied to other tasks in the field of space science, such as aurora image classification, aurora intensity modeling, aurora event detection based on space physics, and aurora image retrieval models based on saliency weights using the Mask RCNN. Therefore, some scholars began to use LSTM networks or C3D networks to extract image features of auroral substorm sequences, thereby realizing automated auroral substorm recognition.

[0003] The best existing auroral substorm recognition models based on LSTM networks and C3D networks have good performance in automatically recognizing substorms, but they cannot establish long-term inter-frame relationships, and there is a large gap between the substorm sequences they recognize and the auroral substorm sequences recognized by space physics experts. These models simply apply existing deep learning networks directly to auroral substorm recognition, without considering the particularity of their images and the underlying space physics knowledge. At the same time, some Transformer-based image or video classification or recognition models have very good performance in natural image tasks, but because auroral substorm images are very different from natural images, they are not suitable for or cannot be directly used in auroral substorm recognition tasks. There is a lack of dedicated deep learning networks for auroral substorm sequence recognition. When existing deep learning networks are directly used to recognize auroral substorm sequences, their accuracy is limited.

[0004] Patent document with application number CN201710937041.7 discloses "an image retrieval method based on hierarchical convolutional neural network", which mainly solves the problem of low accuracy in existing all-sky aurora image retrieval. Its implementation steps are: (1) using adaptive fence method to determine local key points of all-sky image; (2) extracting local SIFT features of all-sky aurora image and constructing visual dictionary; (3) pre-training and fine-tuning convolutional neural network and constructing polarization area pooling layer; (4) extracting regional CNN features of all-sky aurora image; (6) constructing inverted index table and separately saving global CNN features; (7) extracting sub-volume features of query image and calculating its similarity with database image, and outputting retrieval results. This method uses sub-volume features to achieve matching between local key points, solving the problem of high false alarm rate in existing image retrieval methods. However, since it is mainly aimed at all-sky aurora images and is an image classification method, it cannot identify substorm events based on ultraviolet aurora image sequence.

[0005] The patent document with application number CN202311569468.8 discloses a method for retrieval of auroral substorm images based on cross-modality of space environment parameters, and its implementation steps are: generating a substorm event training set consisting of space environment parameters and auroral images; constructing a deep supervised cross-modal retrieval network with a cross-attention mechanism; training the cross-modal retrieval network; and retrieving auroral substorm images using space environment parameters. This method trains the network by using a multi-head cross-attention mechanism and a deep supervised learning algorithm to avoid interference caused by redundant information of global features, extract more accurate image features and parameter features; and retrieve auroral substorm images by using space physics parameter information. However, since it mainly retrieves substorm images collected by the SUSSI satellite, the designed deep learning network does not consider the physical characteristics of auroral images and the experience knowledge of space physics experts, and the network also requires corresponding space physics parameters as input, so it cannot directly recognize ultraviolet aurora image sequences. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose an automated auroral substorm recognition method based on visual eye movement patterns and deep learning, so as to integrate the physical characteristics of aurora images and the empirical knowledge of space physics experts into the deep learning network, thereby improving the accuracy of substorm event recognition based on ultraviolet aurora image sequences.

[0007] The technical solution to achieve the above purpose includes an aurora substorm identification method and an aurora substorm identification device:

[0008] 1. The aurora substorm recognition method based on visual eye movement patterns and deep learning of the present invention is characterized by comprising:

[0009] Construct a space physics expert eye movement dataset, which contains the original ultraviolet aurora image sequence AS and some expert eye movement gaze pattern sequence ES;

[0010] Read each aurora image in the sequence AS I And its corresponding expert eye movement pattern diagram ES I As the initial training set, each aurora image AS in the remaining sequence I As an initial test set;

[0011] Construct a visual eye movement pattern prediction module including feature extraction and prediction head, and use the sum of cross entropy and linear correlation coefficient as the loss function L1 of this module;

[0012] A substorm sequence recognition module was constructed, which included magnetic local time-geomagnetic latitude segmentation, feature embedding, feature encoding, and visual pattern embedding of the substorm recognition head. The cross entropy function was used as the loss function L2 of the module.

[0013] The eye movement pattern prediction module is trained with the initial training set until the loss function converges, and the initial test set is input into the trained eye movement pattern prediction module to obtain its corresponding eye movement pattern prediction graph EP I , and then the aurora images in the initial test set AS I And its corresponding eye movement pattern prediction map EP I Divided into final training set and final test set;

[0014] The final training set is used to train the visual pattern embedded substorm sequence recognition module until the loss function converges, and the final test set is input into the trained visual pattern embedded substorm sequence recognition module to obtain the sequence recognition result.

[0015] 2. The aurora substorm recognition device based on visual eye movement patterns and deep learning of the present invention is characterized by comprising:

[0016] The space physics expert eye movement data collection module 1 is used to collect the eye movement data of the experts when they watch the auroral substorm sequence, and transmit the collected data to the expert eye movement data set construction module 2;

[0017] An expert eye movement data set construction module 2 is used to construct an initial training set and a test set, and use the initial training set to train a visual eye movement pattern prediction module 3;

[0018] A visual eye movement pattern prediction module 3, used for outputting an eye movement gaze pattern prediction map corresponding to the aurora image;

[0019] The substorm recognition data set construction module 4 is used to construct the final training set and test set; the substorm sequence recognition module 5 embedded with visual patterns is trained and tested using the final training set;

[0020] The visual pattern embedded substorm sequence recognition module 5 is used to output the recognition result of the aurora substorm sequence.

[0021] Compared with the prior art, the present invention has the following effects:

[0022] First, use the visualization of expert experience to help improve model performance.

[0023] Existing auroral substorm identification methods directly use existing models or algorithms in the field of machine learning to extract features of auroral images. They all ignore the physical properties of auroral images and the empirical knowledge of space physics experts.

[0024] In the present invention, eye movement patterns are considered as a kind of empirical knowledge. By collecting eye movement data from multiple space physicists, an auroral substorm eye movement dataset is established. By analyzing these eye movement data, expert eye movement gaze pattern diagrams of various types of auroral substorms are generated, which are intuitive and easy to input into deep learning networks, helping to improve model performance.

[0025] Second, the model design is well-founded and has high recognition accuracy.

[0026] The present invention learns the subjective visual gaze patterns of experts in the visual eye movement pattern prediction module and incorporates scientific knowledge in the field of space physics into the substorm sequence recognition module embedded in the visual pattern. That is, a deep learning model suitable for auroral substorm sequence recognition is designed by combining subjective and objective aspects. It not only provides an enhanced representation of auroral substorm characteristics, but also effectively improves the accuracy of ultraviolet auroral substorm image sequence recognition and the rationality of model design. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the implementation of the present invention;

[0028] Figure 2 It is a sub-flow chart of expert eye movement data collection in the present invention;

[0029] Figure 3 is a schematic diagram of the expert visual gaze pattern in the present invention;

[0030] Figure 4 is a structural diagram of a visual eye movement pattern prediction module in the present invention;

[0031] Figure 5 This is the structure diagram of the substorm sequence recognition module embedded in the visual pattern of the present invention

[0032] Figure 6 It is a schematic diagram of magnetic local time-geomagnetic latitude blocks in the present invention;

[0033] Figure 7It is a schematic diagram of the feature embedding submodule in the present invention;

[0034] Figure 8 It is a structural diagram of the feature encoding submodule and the recognition head submodule in the present invention;

[0035] Fig. 9 It is a structural block diagram of the device of the present invention;

[0036] Fig.10 It is a subjective comparison diagram of the eye movement gaze pattern prediction results generated by the present invention and the existing method. DETAILED DESCRIPTION

[0037] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0038] One of the key technologies of this example is: setting up two modules: a visual eye movement pattern prediction module and a visual pattern embedded substorm sequence recognition module. The visual eye movement pattern prediction module aims to learn and imitate the real expert eye movement patterns, and is trained using the eye movement data of experts observing auroral substorm sequences; the visual pattern embedded substorm sequence recognition module is used to embed visual patterns and spatial physics knowledge into the deep learning network for substorm event recognition. The second key technology is: considering the temporal nature of substorm events, the best model transformer for processing time data is used as the network backbone, thereby improving the accuracy of the automatic auroral substorm recognition method and increasing the rationality of the model design.

[0039] Example 1: Auroral substorm recognition method based on visual eye movement patterns and deep learning

[0040] Reference Figure 1 The implementation steps of this example include the following:

[0041] Step 1: Collect eye movement data of spatial physics experts.

[0042] Reference Figure 2 , the implementation of this step is as follows:

[0043] 1.1) The expert sits 50-100 cm away from the screen of the eye tracker, fixes his head, and calibrates the eyeballs using the 9-point calibration method; according to the introduction displayed on the screen, press the button to start playing the aurora image sequence;

[0044] 1.2) After the expert has watched the current sequence, he / she records the judgment result in the experiment record book, and the experiment ends; if the expert cannot give a judgment result of the sequence after watching this time, he / she returns to step 1.1) to watch again and record the judgment result, and the experiment ends;

[0045] 1.3) Export the expert's gaze eye movement information file, which contains the expert's gaze point location and the duration of each gaze point;

[0046] In this example, a total of 7 experts participated in the collection of eye movement data. Each expert repeated steps 1.1)-1.3) until the collection of eye movement data of all experts was completed.

[0047] Step 2: Convert the expert eye movement data into an expert eye movement fixation pattern diagram.

[0048] In order to utilize the expert’s visual eye movement information, it is necessary to convert the eye movement data from two-dimensional points into visual representation, that is, to map the collected expert gaze position coordinates to the auroral substorm image AS. I On a matrix of the same size, the expert eye movement pattern map ES is generated I , complete the conversion from two-dimensional points to visual representation, its implementation includes the following:

[0049] 2.1) First generate an all-zero matrix of the same size as the aurora image,

[0050] 2.2) Mark the expert's eye gaze point on the all-zero matrix, and set the pixel value of the expert's eye gaze position to 255. Then, with this position as the center, assign the position within the circle with a radius of 10 to 255;

[0051] 2.3) Repeat 2.2) until all the expert gaze points on the aurora image are marked in the all-zero matrix, and obtain the eye movement gaze pattern diagram of the aurora image;

[0052] 2.4) Repeat 2.1)-2.3) to generate the corresponding expert eye movement pattern map for all auroral images in each substorm sequence, such as Figure 3 As shown. Figure 3 The first row is a substorm image sequence, and the second row is its corresponding expert eye movement pattern map ES I .

[0053] Step 3: Build a visual eye movement pattern prediction module.

[0054] This example uses a simple network built on the Transformer architecture to build a visual eye movement pattern prediction module to generate an eye movement pattern prediction graph EP I .

[0055] Reference Figure 4 , the visual eye movement pattern prediction module includes: a feature extraction submodule and a prediction head submodule, and the structure of each submodule is as follows:

[0056] The feature extraction submodule is composed of a ResNet50 network and a visual converter ViT network in series, which is used to extract image features, namely the aurora image AS I Extracting primary image features through ResNet50 model I R , the primary image feature I R Extracting High-Level Image Features via Visual Transformer ViT Network I V , size is 4096*512;

[0057] The prediction head submodule is composed of a latitude conversion volume, a linear transformation volume, a normalization volume, a two-dimensional convolution volume and an upsampling volume in series, which is used to generate an eye movement gaze pattern prediction map, that is, to extract high-level image features I from the ViT network. V Input the module, and the feature size after each volume changes to: 512*4096, 512*64*64, 512*64*64, 1*64*64 and 1*241*241. The last two dimensions of the last volume are taken as the final generated eye movement pattern prediction map EP. I .

[0058] Step 4: Set the loss function L1 of the visual eye movement pattern prediction module.

[0059] Calculate the correlation coefficient: CC(EP I , E.S. I )=cov(EP I , E.S. I ) / δ ES *δ EP , where cov() represents the covariance function, δ ES and δ EP They represent the expert eye movement pattern diagram ES I and eye movement pattern prediction map EP I The standard deviation of

[0060] Calculate the cross entropy: Where i represents the expert eye gaze pattern map ES I and eye movement pattern prediction map EP I Pixels in ;

[0061] The loss function L1 of the visual eye movement pattern prediction module is defined based on the calculated correlation coefficient and cross entropy:

[0062] L1=CC(EP I , E.S. I )+H(EP I , E.S. I ).

[0063] Step 5: Use the initial training set to train the eye movement pattern prediction module.

[0064] In order to facilitate the network to better distinguish and learn the eye movement patterns at different stages of the substorm image sequence, this module does not consider the temporal nature of the sequence. Its training steps include:

[0065] 5.1) Read the aurora images AS in the initial training set respectively I And its corresponding expert eye movement pattern diagram ES I ;

[0066] 5.2) The expert eye gaze pattern graph ES I Convert to grayscale image and convert the aurora image AS in the initial training set I The corresponding expert eye gaze pattern diagram ES I Convert from BGR format to RGB format and then to tensor;

[0067] 5.3) Set the size of the aurora sequence images in the initial dataset to 241*241*3, the size of the corresponding expert eye movement pattern map to 241*241, the batch size to 1, the optimizer to use SGD, the number of training rounds to 400, the initial learning rate to 0.003, and the momentum to 0.9;

[0068] 5.4) Input a batch of aurora images into the eye movement pattern prediction module to obtain the eye movement pattern prediction map of the batch; calculate the loss value using the eye movement pattern prediction map and the expert eye movement pattern map according to the loss function L1, and use the SGD optimizer to back-propagate the loss value and update the network parameters;

[0069] 5.5) Repeat step 5.4), reduce the learning rate by 10 times every 20 iterations, until the maximum number of training rounds is 400 or the loss function stops decreasing within 10 iterations, and then stop training to obtain a trained eye movement pattern prediction module.

[0070] Step 6: Use the initial test set to generate an eye movement pattern prediction map and divide it into a final training set and a final test set.

[0071] 6.1) Read the aurora images AS in the initial test set respectively I , and convert it from BGR format to RGB format and then convert it into a tensor;

[0072] 6.2) All aurora images in the test set are AS I Input into the trained eye movement pattern prediction module in sequence to obtain the corresponding eye movement pattern prediction graph EP I ;

[0073] 6.3) Transform the above aurora image AS I and eye movement pattern prediction map EP IThey are arranged in sequence according to their timestamps, and then the eye movement pattern prediction map sequence and all aurora image sequences are divided into a final training set and a final test set in a ratio of 2:1.

[0074] Step 7: construct a substorm sequence recognition module embedded with visual patterns to identify whether the input aurora sequence is a substorm sequence.

[0075] Physicists usually convert auroral substorm images into the magnetic local time (MLT) and geomagnetic latitude (MLAT) coordinate systems for analysis. In this coordinate system, the dynamic changes of bright spots on the auroral oval can be observed with a unified standard. Therefore, when dividing and encoding, it is necessary to process them separately in these two directions.

[0076] Reference Figure 5 ,The substorm sequence recognition module embedded by visual pattern ,constructed in this example includes: a magnetic local time - geomagnetic latitude block ,submodule, a feature embedding submodule, a feature encoding submodule and a substorm recognition ,header submodule.

[0077] The magnetic local time and geomagnetic latitude block submodule is a new image block method, such as Figure 6 As shown in FIG. 1 , it is composed of the magnetic local time MLT block and the geomagnetic latitude MLAT block in parallel, wherein the magnetic local time block is used to divide the image into four main areas according to the magnetic local time, representing noon, morning, dusk and night, respectively, to obtain the magnetic local time block E. MLT ;

[0078] The geomagnetic latitude segmentation is based on the fact that auroral substorms often occur between 50 and 90 degrees. When segmenting the geomagnetic latitude, the image is segmented into concentric circles, that is, a block is taken every 10 degrees from 50 to 90 degrees to obtain different geomagnetic latitude blocks, including the geomagnetic latitude block E within the range of 80-90 degrees. MLAT89 , geomagnetic latitude block E in the range of 70-80 degrees MLAT78 , geomagnetic latitude block E in the range of 60-70 degrees MLAT67 , geomagnetic latitude block E in the range of 50-60 degrees MLAT56 ;

[0079] The feature embedding submodule consists of a flattened volume, a concatenated volume, two normalized volumes, a fully connected volume, a positional encoding in series, and then a positional encoding in parallel, such as Figure 7 shown.

[0080] The concatenation part is used to change the dimension of the geomagnetic latitude block to the same size as the magnetic local time block and perform feature embedding. Assuming that the dimensions of each vector are d1, d2, d3, and d4, the four image block vectors are concatenated through the concat operation to obtain an image block vector with a dimension of 4*(d1+d2+d3+d4); after two normalized volumes and one fully connected volume, the feature of an aurora image after geomagnetic latitude embedding can be obtained, and its dimension size is 4*1024. The position code is added to the feature to obtain a geomagnetic latitude feature E with a size of 5*1024. MLAT .

[0081] The parallel part is used to encode the position of the magnetic local time block, and the magnetic local time feature E with a size of 5*1024 is also obtained. MLT ;

[0082] The feature encoding submodule consists of two sets of Token allocation, Transformer encoder, feature concatenation volume, maximum pooling volume, and Transformer encoder in parallel and then in series with a feature concatenation volume, such as Figure 8 shown.

[0083] This module is used to encode the embedded features of the aurora substorm sequence and the embedded features of the gaze pattern prediction map sequence to obtain the fusion feature T AV , its specific implementation includes the following:

[0084] For each aurora image sequence and its corresponding eye gaze pattern prediction map sequence, after magnetic local time and geomagnetic latitude block segmentation and feature embedding, the magnetic local time and geomagnetic latitude embedding features are obtained respectively: Magnetic local time embedding features of aurora substorm sequence Embedded features of geomagnetic latitude in auroral substorm sequences Magnetic local time embedding features of gaze prediction graph sequences Geomagnetic latitude embedding features of the gaze prediction map sequence The size of all features is expressed as (b*f*p*d), where b represents the batch size; f is the number of frames or sub-block size; p represents the number of blocks + 1, 1 represents the position encoding; d is the embedding dimension;

[0085] Embedding characteristics of magnetic local time in auroral substorm sequences Embedded features of geomagnetic latitude in auroral substorm sequences Magnetic local time embedding features of gaze prediction graph sequences Geomagnetic latitude embedding features of the gaze prediction map sequence The corresponding tokens are allocated respectively and are expressed as: and

[0086] The Transformer encoder is used to encode the Tokens assigned to the aurora image sequence embedding features to obtain the aurora substorm sequence feature T AS (10*5*2048); that is, embedding characteristics of magnetic local time Assigned and geomagnetic latitude embedding features Assigned Perform attention calculation and then splice frame by frame and Get the Token feature T of the auroral substorm sequence AS (10*5*2048);

[0087] The Token assigned to the gaze pattern prediction image sequence embedding feature is encoded through the Transformer encoder to obtain the gaze prediction image sequence feature T VS (10*5*2048), that is, first embed the features of magnetic local time Assigned and geomagnetic latitude embedding features Assigned (10*5*1024) for attention calculation; then splicing frame by frame and Get the Token feature T of the gaze prediction graph sequence VS (10*5*2048);

[0088] The auroral substorm sequence characteristics T AS (10*5*2048) and the gaze prediction map sequence feature T VS (10*5*2048) is fused, that is, a maximum pooling volume and Transformer encoder are used to reduce its feature dimension so that T AS The dimension becomes (10*3*1024), T VS The dimension of becomes (10*3*128); then T AS (10*3*1024) and T VS (10*3*128) performs concatenation operation on the second dimension to obtain the fused feature T AV (10*3*1152).

[0089] The substorm recognition head module is composed of an average pooling volume, a Transformer encoder and a multi-volume perceptron in series, such as Figure 8As shown, it is used to fusion feature T AV Encode and classify, that is, the fused feature T AV (10*3*1152) is input into the recognition head to obtain the final recognition result; the feature size changes after each roll are: 10*3*1152, 10*1*576, 10*576, 10*2.

[0090] Step 8: According to the substorm sequence recognition results and the true sequence labels, the loss function L2 of the substorm sequence recognition module embedded with visual patterns is set.

[0091]

[0092] Among them, Q j represents the true label of the sequence, P j represents the substorm sequence recognition result, j represents the sequence label Q j and substorm sequence identification results P j The number of sequences.

[0093] Step 9: Process the aurora image sequences and eye gaze pattern prediction map sequences in the final training set and the final test set into sequences S of equal length p :

[0094]

[0095] Among them, p and k represent the sequence lengths before and after processing respectively, and L is the set sequence length, which is set to 10 in this example; finally, each training sample includes an auroral substorm image sequence and a corresponding eye movement pattern prediction map arranged in order.

[0096] Step 10: Use the final training set to train the substorm sequence recognition module embedded with visual patterns.

[0097] 10.1) Read all images in the eye movement pattern prediction image sequence in the final training set, convert them into grayscale images and then convert them into tensors; read all images in the aurora sequence in the final training set, convert them from BGR format to RGB format and then convert them into tensors;

[0098] 10.2) Set the size of the aurora sequence in the initial data set to 10*241*241, the batch size to 10, the optimizer to use Adam, the number of training rounds to 100, the initial learning rate to 0.001, and the momentum to 0.9;

[0099] 10.3) Input a batch of aurora sequences and eye movement pattern prediction map sequences into the substorm sequence recognition module of visual pattern embedding to obtain the sequence recognition results of the batch; use the recognition results and the true labels to calculate the loss value according to the loss function L2, and use the Adam optimizer to back-propagate the loss value and update the network parameters;

[0100] 10.4) Repeat step 10.3), and reduce the learning rate by 10 times every 10 iterations until the maximum number of training rounds reaches 100 or the loss function no longer decreases within 10 iterations, then the training stops and the substorm sequence recognition module embedded with the trained visual pattern is obtained;

[0101] Step 11: Use the final test set to obtain the auroral substorm identification results.

[0102] Read all the images in the aurora sequence in the final test set, convert them from BGR format to RGB format and then convert them into tensors;

[0103] Then it is input into the substorm sequence recognition module embedded with the trained visual pattern to obtain the recognition result of whether the sequence is a substorm sequence.

[0104] Example 2: Aurora substorm recognition device based on visual eye movement patterns and deep learning

[0105] Reference Fig. 9 ,The aurora substorm recognition device in this example includes: a space physics expert eye movement data collection module 1, an expert eye movement data set building module 2, a visual eye movement pattern prediction module 3, a substorm recognition data set building module 4, a visual pattern embedded substorm sequence recognition module 5;

[0106] The space physics expert eye movement data collection module 1 is used to collect the eye movement data of the experts when they watch the aurora substorm sequence, and the collected data is transmitted to the expert eye movement data set construction module 2;

[0107] The expert eye movement data set construction module 2 includes an expert eye movement gaze pattern map generation submodule 21 and a data division submodule 22; wherein the eye movement gaze pattern map generation submodule 21 is used to generate an expert eye movement gaze pattern map using the collected expert eye movement gaze data; the data division submodule 22 is used to divide the aurora image and its corresponding expert eye movement gaze pattern map into an initial training set and a test set; the initial training set and the test set are input into the visual eye movement pattern prediction module 3 for module training;

[0108] The visual eye movement pattern prediction module 3 includes: a feature extraction submodule 31 and a prediction head submodule 32; wherein the feature extraction submodule 31 is used to extract aurora image features, and the extracted aurora image features are input into the prediction head submodule 32 to generate an eye movement gaze pattern prediction map; the eye movement gaze pattern prediction map sequence output by the aurora image sequence in the initial test is input into the substorm recognition data set construction module 4;

[0109] The substorm recognition data set construction module 4 is used to construct the final training set and test set, which contain aurora image sequences, eye movement pattern prediction map sequences and sequence true recognition labels, for training and testing the substorm sequence recognition module 5 embedded with visual patterns;

[0110] The substorm sequence recognition module 5 embedded with visual patterns comprises: a magnetic local time-geomagnetic latitude segmentation submodule 51, a feature embedding submodule 52, a feature encoding submodule 53 and a substorm recognition head submodule 54; wherein the magnetic local time-geomagnetic latitude segmentation submodule 51 is used to segment the aurora image sequence and the eye movement gaze pattern prediction map sequence according to their spatial physical properties; the feature embedding submodule 52 is used to embed features into the magnetic local time-geomagnetic latitude blocks of the aurora images and the magnetic local time-geomagnetic latitude blocks of the eye movement gaze pattern prediction maps; the embedded features are used to be input into the feature encoding submodule 53 for feature encoding and fusion; the fused features are used to be input into the substorm recognition head submodule 54 to output the recognition results of the aurora substorm sequence.

[0111] The effects of the present invention are further described in detail below in conjunction with simulation experiments.

[0112] 1. Simulation experiment conditions

[0113] The computer processor used is Intel(R) Core(TM) i5-6600 CPU@3.30GHz, 16GB of memory, and the graphics card is two NVIDIA RTX 3090GPUs with 24GB of video memory.

[0114] The operating system is 64-bit Ubuntu 18.04, the algorithm simulation adopts Python language, and the deep learning framework PyTorch version 2.1.0 is used.

[0115] The evaluation indicators are the common accuracy, precision, recall and F-score in behavior recognition, among which:

[0116] Accuracy represents the proportion of all correctly predicted categories in the total number, and is calculated as follows:

[0117]

[0118] Where TP is the number of positive samples predicted as positive samples, FN is the number of positive samples predicted as negative samples, FP is the number of negative samples predicted as positive samples, and TN is the number of negative samples predicted as negative samples;

[0119] The precision rate Precision represents the proportion of positive samples in the prediction pair as positive samples in the entire prediction, which is calculated as follows:

[0120]

[0121] The recall rate Recall represents the proportion of the predicted positive samples in the entire true positive samples, which is calculated as follows:

[0122]

[0123] F-score is a unified indicator for measuring the recall and precision of the model, and is calculated as follows:

[0124]

[0125] When β is 1, the F-score is the commonly used F1.

[0126] 2. Simulation experiment content and results

[0127] Experiment 1: Under the above simulation experimental conditions, the eye movement fixation point prediction method of the present invention and the existing one was used to train on the initial training set collected in step 1, and the 58 aurora image sequences in the initial test set were used for testing to obtain the subjective results of eye movement fixation pattern prediction, such as Fig.10 shown.

[0128] from Fig.10 It can be seen that compared with other methods, the eye movement pattern prediction module of the present invention has a smaller gaze point prediction area, a more accurate gaze point position, and is closer to the expert's eye movement gaze pattern diagram.

[0129] Experiment 2: Under the above simulation experimental conditions, the substorm sequence recognition module embedded with the visual pattern of the present invention and the existing six substorm image sequence recognition methods were used for training and testing respectively to obtain sequence recognition results, and the results were evaluated using commonly used classification evaluation indicators. The results are shown in Table 1:

[0130] Table 1 Comparison results of indicators between the present invention and six existing methods on substorm sequence recognition tasks

[0131] method Accuracy↑ Precision↑ Recall↑ F1-score↑ Vit-3d 0.9134 0.8846 0.92 0.9020 Video-Swin-tiny 0.8304 0.8 0.81 0.8050 Video-Swin-small 0.6391 0.5702 0.75 0.6479 Video-FocalNet 0.9091 0.8692 0.93 0.8986 DualFormer-tiny 0.8609 0.8384 0.83 0.8342 Yang's - 0.4928 0.9198 0.6417 EMSF-R2Plus1D 0.8826 0.8462 0.88 0.8627 EMSF-C3D 0.9087 0.8911 0.9 0.8955 The present invention 0.9264 0.9029 0.93 0.9163

[0132] It can be seen from Table 1 that the substorm sequence recognition module embedded with visual patterns of the present invention is superior to the existing methods in terms of the above indicators, indicating that the present invention can better recognize substorm sequences.

[0133] The above simulation results show that the present invention has higher accuracy and robustness in identifying space physical events.

[0134] The above 6 comparison methods are from the following sources:

[0135] Vit-3d is a method in the paper titled An image is worth 16x16 words: Transformers for image recognition at scale published on the arXiv website in 2020;

[0136] Video-Swin-tiny and Video-Swin-small are methods in the paper titled Video SwinTransformer published on the arXiv website;

[0137] Video-FocalNet is a method published in the 2023 Proceedings of the IEEE International Conference on Computer Vision, titled Video-FocalNets: Spatio-TemporalFocal Modulation for Video Action Recognition.

[0138] DualFormer-tiny is a method in the paper titled DualFormer: Local-Global Stratified Transformer for Efficient Video Recognition published in the Proceedings of the IEEE International Conference on Computer Vision in 2023;

[0139] Yang's is the method in the paper Improving level set method for fast auroral oval segmentation published in the IEEE Transaction on Image Processing journal in 2014;

[0140] EMSF-R2Plus1D and EMSF-C3D are methods published in the paper titled Progressively Coding the Human Visual Information for Auroral Substorm Recognition at the International Conference on Frontiers of Electronics, Information and Computation Technologies in 2023.

[0141] The above description is only a specific example of the present invention and does not constitute any limitation to the present invention. It shows that for professionals in this field, after understanding the content and principles of the present invention, it is possible to make various modifications and changes in form and details without deviating from the principles and structures of the present invention. For example, the eye movement data processing method of the present invention, in addition to visualizing the expert gaze point as an expert eye movement gaze pattern diagram in this example, can be adjusted for different types of data; the feature extractors involved in the present invention, except for Resent50-ViT used in this example, can be replaced by other feature extraction modules. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0142] It should be noted that the step numbers in the specification and claims of the present invention are only for the purpose of clearly describing the embodiments of the present invention and facilitating understanding, and the sequence of the step numbers is not limited.

Claims

1. An automated auroral substorm recognition method based on visual eye movement patterns and deep learning, characterized in that: include: Construct a space physics expert eye movement dataset, which contains the original ultraviolet aurora image sequence AS and some expert eye movement gaze pattern sequence ES; Read each aurora image in the sequence AS I and its corresponding expert eye gaze pattern map ES, as the initial training set, each aurora image AS in the remaining sequence I As an initial test set; Construct a visual eye movement pattern prediction module including feature extraction and prediction head, and use the sum of cross entropy and linear correlation coefficient as the loss function L1 of this module; A substorm sequence recognition module was constructed, which included magnetic local time-geomagnetic latitude segmentation, feature embedding, feature encoding, and visual pattern embedding of the substorm recognition head. The cross entropy function was used as the loss function L2 of the module. The eye movement pattern prediction module is trained with the initial training set until the loss function converges, and the initial test set is input into the trained eye movement pattern prediction module to obtain its corresponding eye movement pattern prediction graph EP I , and then the aurora images in the initial test set AS I And its corresponding eye movement pattern prediction map EP I Divided into final training set and final test set; The final training set is used to train the visual pattern embedded substorm sequence recognition module until the loss function converges, and the final test set is input into the trained visual pattern embedded substorm sequence recognition module to obtain the sequence recognition result.

2. The method according to claim 1, characterized in that The construction of the spatial physics expert eye movement dataset includes: 2a) Collecting expert eye movement data: 2a1) The expert sits 50-100 cm away from the screen of the eye tracker, fixes his head, and calibrates the eyeballs using the 9-point calibration method; 2a2) According to the introduction displayed on the screen, press the button to start playing the aurora image sequence; 2a3) After the expert has watched the current sequence, he / she records the judgment result in the experimental record book, and the experiment ends; if the expert cannot give a judgment result of the sequence after watching this time, repeat steps 2a1)-2a2) to watch again and record the judgment result, and the experiment ends; 2a4) exporting the expert's gaze eye movement information file, which includes the expert's gaze point position and the duration of each gaze point; 2b) Convert the eye movement data into an expert eye movement pattern map ES I : 2b1) generating an all-zero matrix of the same size as the aurora image, and setting the pixel value of the position where the expert's eye movement is fixated to 255 in the all-zero matrix; 262) Taking each fixation point as the center of the circle, assign all pixels in the circle with a radius of 10 to 255, and obtain the expert eye movement fixation pattern diagram of the aurora image; 2b3) For all images in each aurora sequence, repeat steps 2b1) to 2b2) to obtain all aurora images AS I Generate the corresponding expert eye gaze pattern map ES I .

3. The method according to claim 1, characterized in that The visual eye movement pattern prediction module includes: a feature extraction submodule and a prediction head submodule: The feature extraction submodule consists of a ResNet50 network connected in series with a visual converter ViT network, which is used to extract image features, namely the aurora image AS I Extracting primary image features through ResNet50 model I R , the primary image feature I R Extracting High-Level Image Features via Visual Transformer ViT Network I V ; The prediction head submodule is composed of a latitude conversion layer, a linear transformation layer, a normalization layer, a two-dimensional convolution layer and an upsampling layer in series, which is used to convert the high-level features I V Input the prediction head submodule to get the final generated eye movement pattern prediction map EP I。 4. The method according to claim 1, characterized in that: The loss function L1 of the visual eye movement pattern prediction module is expressed as follows: L1=CC(EP I ,IS I )+H(EP I ,IS I ) Among them, CC(EP I , E.S. I )=cov(EP I , E.S. I ) / δ ES *δ EP is the correlation coefficient, cov() represents the covariance function, δ ES and δ EP They represent the expert eye movement pattern diagram ES I and eye movement pattern prediction map EP I The standard deviation of is the cross entropy, i represents the expert eye movement pattern graph ES I and eye movement pattern prediction map EP I The pixels in .

5. The method according to claim 1, characterized in that: The substorm sequence recognition module embedded in the visual pattern includes: a magnetic local time-geomagnetic latitude block submodule, a feature embedding submodule, a feature encoding submodule and a substorm recognition header submodule: The magnetic local time and geomagnetic latitude block submodule is composed of a magnetic local time MLT block and a geomagnetic latitude MLAT block in parallel. The magnetic local time block is used to divide the image into four main areas according to the magnetic local time, representing noon, morning, dusk and night, respectively, to obtain the magnetic local time block E MLT ; Geomagnetic latitude blocks are used to block the image in concentric circles, taking a block every 10 degrees from 50 degrees to 90 degrees to obtain different geomagnetic latitude blocks E MLAT89 , E MLAT78 , E MLAT67 , E MLAT56 ; The feature embedding submodule consists of a flattening layer, a concatenation layer, two normalization layers, a fully connected layer, a position code connected in series, and then a position code connected in parallel; the series connection part is used to convert the dimension of the geomagnetic latitude block into the same dimension as the magnetic local time block E MLT The same size and feature embedding; the parallel part is used to embed the features of the magnetic local time block; the magnetic local time features E are obtained respectively MLT and geomagnetic latitude characteristics E MLAT ; The feature encoding submodule consists of two groups of Token allocation, Transformer encoder, feature concatenation layer, maximum pooling layer, and a feature concatenation layer connected in parallel with the Transformer encoder. It is used to encode the embedded features of the aurora substorm sequence and the embedded features of the gaze pattern prediction map sequence to obtain the fused feature T AV ; The substorm recognition head module consists of an average pooling layer, a Transformer encoder and a multi-layer perceptron in series, which is used to transform the fusion feature T AV Encoding and classification are performed to obtain the sequence recognition results.

6. The method according to claim 1, characterized in that The loss function L2 of the substorm sequence recognition module embedded with the visual pattern is expressed as follows: Where j represents the sequence label Q j and substorm sequence identification results P j The number of sequences.

7. The method according to claim 1, characterized in that The step of training the eye movement pattern prediction module with the initial training set comprises: 7a) The expert eye gaze pattern graph ES I Convert to grayscale image; convert the aurora image AS in the initial training set I The corresponding expert eye gaze pattern diagram ES I Convert from BGR format to RGB format and then to tensor; 7b) Set the size of the aurora sequence images in the initial data set to 241*241, the batch size to 1, the optimizer to use SGD, the number of training rounds to 400, the initial learning rate to 0.003, and the momentum to 0.9; 7c) inputting a batch of aurora images into the eye movement pattern prediction module to obtain the eye movement pattern prediction map of the batch; calculating the loss value of the eye movement pattern prediction map and the expert eye movement pattern map according to the loss function L1, and back-propagating the loss value using the SGD optimizer to update the network parameters; 7d) Repeat step 7c), reduce the learning rate by 10 times every 20 iterations, until the maximum number of training rounds reaches 400 or the loss function stops decreasing within 10 iterations, and then stop training to obtain a trained eye movement pattern prediction module.

8. The method according to claim 1, characterized in that The aurora images AS in the initial test set I And its corresponding eye movement pattern prediction map EP I It is divided into a final training set and a final test set, that is, the aurora image sequence in the initial test set is input into the trained eye movement pattern prediction module to obtain its corresponding eye movement fixation pattern prediction map sequence, and then the eye movement fixation pattern prediction map sequence and all the aurora image sequences are divided into a final training set and a final test set in a ratio of about 2:

1.

9. The method according to claim 1, characterized in that: The method of training the substorm sequence recognition module embedded in the visual pattern with the final training set includes: 9a) Process the aurora image sequences and eye gaze pattern prediction map sequences in the final training set and the final test set into sequences S of equal length p : Among them, p and k represent the sequence lengths before and after processing, respectively, and L is the set sequence length; 9b) The processed eye movement pattern prediction image sequence S p Convert to grayscale images; read all images in the aurora sequence and eye gaze pattern prediction map sequence in the final training set, convert them from BGR format to RGB format and then convert them into tensors; 9c) Set the size of the aurora sequence in the initial data set to 10*241*241, the batch size to 10, the optimizer to use Adam, the number of training rounds to 100, the initial learning rate to 0.001, and the momentum to 0.9; 9d) Inputting a batch of aurora sequences and eye movement pattern prediction map sequences into the substorm sequence recognition module of visual pattern embedding to obtain the sequence recognition results of the batch; using the recognition results and the true labels to calculate the loss value according to the loss function L2, and using the Adam optimizer to back-propagate the loss value to update the network parameters; 9e) Repeat step 9d), and reduce the learning rate by 10 times every 10 iterations until the maximum number of training rounds reaches 100 or the loss function no longer decreases within 10 iterations. The training stops and the substorm sequence recognition module embedded with the trained visual pattern is obtained.

10. An auroral substorm recognition device based on visual eye movement patterns and deep learning, characterized in that: include: The space physics expert eye movement data collection module 1 is used to collect the eye movement data of the experts when they watch the auroral substorm sequence, and transmit the collected data to the expert eye movement data set construction module 2; An expert eye movement data set construction module 2 is used to construct an initial training set and a test set, and use the initial training set to train a visual eye movement pattern prediction module 3; A visual eye movement pattern prediction module 3, used for outputting an eye movement gaze pattern prediction map corresponding to the aurora image; The substorm recognition data set construction module 4 is used to construct the final training set and test set; the substorm sequence recognition module 5 embedded with visual patterns is trained and tested using the final training set; The visual pattern embedded substorm sequence recognition module 5 is used to output the recognition result of the aurora substorm sequence.

11. The device according to claim 10, characterized in that: The expert eye movement data set construction module 2 includes an expert eye movement gaze pattern map generation submodule 21 and a data division submodule 22; wherein the eye movement gaze pattern map generation submodule 21 is used to generate an expert eye movement gaze pattern map using the collected expert eye movement gaze data; the data division submodule 22 is used to divide the aurora images and their corresponding expert eye movement gaze pattern maps into an initial training set and a test set; The visual eye movement pattern prediction module 3 includes a feature extraction submodule 31 and a prediction head submodule 32; wherein the feature extraction submodule 31 is used to extract aurora image features, and the extracted aurora image features are input into the prediction head submodule 32 to generate an eye movement gaze pattern prediction map; The substorm sequence recognition module 5 embedded with visual patterns comprises a magnetic local time-geomagnetic latitude segmentation submodule 51, a feature embedding submodule 52, a feature encoding submodule 53 and a substorm recognition head submodule 54; wherein the magnetic local time-geomagnetic latitude segmentation submodule 51 is used to segment the aurora image sequence and the eye movement gaze pattern prediction map sequence according to their spatial physical properties; the feature embedding submodule 52 is used to embed features into the magnetic local time-geomagnetic latitude blocks of the aurora images and the magnetic local time-geomagnetic latitude blocks of the eye movement gaze pattern prediction maps; the feature encoding submodule 53 is used to feature encode and fuse the embedded features; the feature substorm recognition head submodule 54 is used to discriminate the fused features and output the recognition results of the aurora substorm sequence.

Images