Image retrieval method for aurora substorm based on cross-modal spatial environment parameters

Abstract

This invention discloses a method for retrieving auroral substorm images based on cross-modal spatial environmental parameters, which can be used to retrieve auroral substorm images using spatial environmental parameters. The implementation steps of this invention are: generating a substorm event training set composed of spatial environmental parameters and auroral images; constructing a deep supervised cross-modal retrieval network with a cross-attention mechanism; training the cross-modal retrieval network; and retrieving auroral substorm images using spatial environmental parameters. This invention uses a multi-head cross-attention mechanism and a deep supervised learning algorithm to train the network, which can avoid interference from redundant information in global features and extract more accurate image features and parameter features. This invention uses spatial physical parameter information to retrieve auroral substorm images, and can adjust different combinations of spatial physical parameters to retrieve substorm events, improving retrieval efficiency.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and further relates to a cross-modal auroral substorm image retrieval method based on space environment parameters in the interdisciplinary field of computer vision and space physics. This invention can be used to mine the correlation between auroral substorm images and space environment parameters, extract image features and parameter features, and use the extracted features to calculate the similarity of image-parameter pairs, thereby enabling the retrieval of auroral substorm events using space environment parameters. Technical Background

[0002] Auroral substorms are a manifestation of electromagnetic disturbances in space, releasing significant amounts of solar wind energy stored in the magnetotail. Current auroral event retrieval primarily relies on image-to-image methods, manually calibrating auroral image features and then using these features to locate corresponding auroral events. In practice, due to limitations in the field of view for ground-based imagers and the limited number and orbital constraints of satellite-based imagers, it is difficult to determine the exact nature of each substorm event using an auroral imager. Furthermore, the influence of space environment parameters on structural changes within auroral events is not considered.

[0003] In their paper "Aurora Image Classification and Retrieval Method Based on Deep Hash Algorithm" (Journal of Electronics and Information Technology, Vol. 42, No. 12, 2020, pp. 3029-3036), Chen Changhong et al. proposed a method for retrieving aurora images using a deep hash algorithm. The method's implementation steps are as follows: First, spatial pyramid pooling and power-mean transform are embedded in a convolutional neural network (CNN) to extract multi-scale regional information from aurora images, and the multi-scale features are fused into global features. Second, hash layers are added between fully connected layers, mapping the high-dimensional semantic information best represented by the fully connected layers into compact binary hash codes. Hamming distance is used to measure the similarity between images in a low-dimensional space. Finally, the method introduces a multi-task learning mechanism, using image label information and the similarity information between image pairs to design a loss function, combining the losses of the classification layer and the hash layer as the optimization objective. This method enables better semantic similarity between hash codes to achieve aurora image retrieval. However, this method still has shortcomings. Due to the inherent characteristics of aurora data, using global features of aurora images will result in a large amount of redundant information, leading to low retrieval accuracy and low retrieval efficiency when searching for aurora events.

[0004] Shaanxi Normal University disclosed a method for aurora event recognition and detection from complex aurora observation videos in its patent application, "A Method for Aurora Motion Characterization Based on Unsupervised Deep Optical Flow Network" (Application No.: 202110084981.2, Application Date: 2021.01.21, Publication No.: CN 112785629 A). The method's first step is to calculate the bidirectional optical flow between two adjacent images using an optical flow network. Next, a bidirectional warped image is calculated, and a bidirectional deformation map is inferred from the bidirectional optical flow. A loss function is then constructed using the all-sky aurora image, the warped image, and the bidirectional deformation map to optimize and train the optical flow network. Finally, the trained optical flow network is used to extract the aurora optical flow field from the aurora observation video as a representation of aurora motion. However, this method still has shortcomings. Simply using the aurora optical flow field is insufficient to analyze the changing trends of the aurora. This method only analyzes the single modality of the image and cannot reflect the influence of spatial environmental parameters on aurora structural changes. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art by proposing a method for retrieving auroral substorm images based on cross-modal space environment parameters. This method aims to solve the problem that the large number of space observation devices makes it difficult to quickly find the desired events in massive amounts of observation data.

[0006] To achieve the above objectives, the present invention constructs and preprocesses a multi-source observation dataset of auroral substorm events, including ultraviolet auroral image data acquired by the DMSP (Defense Meteorological Satellite Program) satellite and space environment parameters provided by NASA. Image data preprocessing involves cropping the input image to a size suitable for the network, centered at (70MLAT, 21MLT) coordinates in the geomagnetic coordinate system. Parameter preprocessing involves determining the substorm expansion phase time using the geomagnetic index, calculating the correlation coefficient between the mean auroral intensity and the space environment parameters to apply a time delay to the space environment parameters, and combining the space environment parameters and images to form sample pairs to construct the auroral substorm event dataset. The present invention first constructs a cross-modal retrieval network to extract features from images and parameters. The cross-modal feature fusion subnetwork in this network calculates the similarity between image features and parameter features using a dot product method. This similarity is then used to assign weights to the image features and parameter features, and the weighted combination yields new image features and parameter features. This results in parameter features and image features containing significant information about each other, avoiding the large amount of redundant information caused by the use of global features in existing technologies. Simultaneously, a deep supervised learning subnetwork is constructed within this network to eliminate the differences between image information and parameter information modalities, ensuring that the trained features have significant discriminative power. Training through this network guarantees the alignment of extracted image features and parameter features while preserving the original linguistic information of each modality.

[0007] The specific steps of this invention include the following:

[0008] Step 1: Generate a training set of substorm events consisting of spatial environment parameters and aurora images:

[0009] Select at least 2000 ultraviolet aurora images in the Arctic LBHL band, of which at least 1000 images contain substorm events; select spatial environmental parameters corresponding to the time of each ultraviolet aurora image; preprocess the selected images and parameters respectively, combine each preprocessed image and its corresponding parameter into a sample pair, and label each sample pair with a category label; combine all sample pairs and their corresponding category labels into a training set.

[0010] The preprocessing involves cropping each image to a size of 224×224 centered at coordinates (70MLAT, 21MLT), and then dividing each cropped image into 36 regions to obtain preprocessed image data. The spatial parameters corresponding to each image are grouped, and the interplanetary magnetic field and solar wind parameters are delayed by 14 minutes to obtain preprocessed parameter data.

[0011] Step 2: Construct a deep supervised cross-modal retrieval network with a cross-attention mechanism, consisting of a two-dimensional image feature extraction subnetwork and a one-dimensional parameter feature extraction subnetwork connected in parallel, and then connected in series with a cross-modal feature fusion subnetwork and a deep supervised learning subnetwork.

[0012] Step 3: Train a cross-modal retrieval network with deep supervision and cross-attention mechanism:

[0013] The training set is input into the cross-modal retrieval network in batches. The discriminant loss value between the network output and the class label of the sample is calculated. The discriminant loss value, the similarity recognition loss value of image features and parameter features, and the modality invariance loss value are weighted and summed to obtain the total loss value of the cross-modal retrieval network. The network parameters are iteratively updated through the backpropagation algorithm until the total loss value of the network converges, and the trained cross-modal retrieval network is obtained.

[0014] Step 4: Retrieve aurora substorm images using spatial environment parameters:

[0015] Using the same method as in step 1, the images and spatial environment parameters to be retrieved are preprocessed. The preprocessed images and parameters are then input into the trained cross-modal retrieval network. The feature similarity between each image and each pair of parameters is calculated, and the image with the highest similarity between each pair of parameters is selected as the result of retrieving aurora substorm images.

[0016] Compared with the prior art, the present invention has the following advantages:

[0017] First, because the present invention uses a multi-head cross-attention mechanism to extract and fuse parametric features and image features across modalities, it overcomes the problem of low retrieval accuracy caused by redundant information in the global features of the image as input features in the prior art. This allows the present invention to avoid interference caused by redundant information in the global features and improve the accuracy of cross-modal retrieval.

[0018] Second, because the present invention uses a deep supervised learning sub-network for training, it eliminates the differences between image information and parameter information modes, effectively learns image and parameter features, overcomes the problems of mode differences and information inconsistency in the prior art, and enables the method of the present invention to better capture the essential features of the data, thereby improving the performance and effectiveness of the model.

[0019] Third, because this invention uses spatial physics parameter information to perform cross-modal retrieval of auroral substorm images, it overcomes the problem of existing technologies that only use image data for auroral substorm event retrieval. This allows the invention to fully utilize parameter data to accurately retrieve auroral substorm images, and also to adjust different combinations of spatial physics parameters to analyze the impact of different parameters on auroral substorm events, thus achieving cross-modal retrieval. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0021] Figure 2 This is a schematic diagram of the cross-modal retrieval network constructed and trained in this invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] Reference Figure 1 The implementation steps of the embodiments of the present invention will be further described below.

[0024] Step 1: Generate a training set and a test set for substorm events, consisting of space environment parameters and aurora images.

[0025] Select at least 2000 ultraviolet aurora images in the Arctic LBHL band, including at least 1000 images of substorm events; select spatial environmental parameters corresponding to the time of each ultraviolet aurora image; preprocess the selected images and parameters respectively, combine each preprocessed image and its corresponding parameter into a sample pair, and label each sample pair with a category label; combine all sample pairs and their corresponding category labels into a training set.

[0026] The preprocessing steps are as follows:

[0027] The first step is to crop each selected image to a size of 224×224 with the coordinates (70MLAT, 21MLT) as the center, and then divide each cropped image into 36 regions to obtain the preprocessed image data.

[0028] The second step involves dividing all space parameters into four groups based on interplanetary magnetic field, solar wind parameters, substorm index, and geomagnetic index. The interplanetary magnetic field includes its three components Bx, By, and Bz; the solar wind parameters include solar wind velocity Vp and solar wind density Np; the substorm index includes the AE index; and the geomagnetic index includes the SYM-H index. Given that the temporal resolution of the space parameters is 1 minute, the parameters are read over a 20-minute time span, resulting in a 20×n dimension for each group, where n represents the number of different space parameters in each group. Finally, the interplanetary magnetic field and solar wind parameters in each selected space environment parameter are delayed by 14 minutes to obtain the preprocessed parameter data.

[0029] The image data in this embodiment of the invention consists of Arctic LBHL-band ultraviolet aurora images captured by the Special Sensor Ultraviolet Spectrographic Imager (SSUSI) aboard the DMSP satellite during the winters of 2005-2010 and 2013-2018. These images were then labeled to indicate whether they were substorm events. Since substorms generally occur on the night side of the Earth, and appear to be in the magnetic local time (MRT) range of 18 MLAT-00 MLAT from the aurora images, each image was cropped to a size of 224 × 224 pixels in the geomagnetic coordinate system, centered at (70 MLAT, 21 MLAT). Each cropped image was then divided into 36 blocks to obtain the preprocessed image data.

[0030] In this embodiment of the invention, the space environment parameter data are selected from the interplanetary magnetic field three components (Bx, By, Bz), solar wind speed (Vp), solar wind density (Np), auroral current index AE, and SYM-H index provided by NASA (National Aeronautics and Space Administration). The space environment parameters at the corresponding time are selected based on the marked auroral substorm image data, and the interplanetary magnetic field three components, solar wind speed, and solar wind density are delayed by 14 minutes to obtain the preprocessed parameter data.

[0031] The preprocessed image data and their corresponding spatial environment parameters were grouped into multiple sample pairs. Sample pairs that were substorm events were labeled with 1, and sample pairs that were not substorm events were labeled with 0. A total of 1032 aurora substorm events were labeled in this dataset. Each group of aurora substorm sample pairs was divided into a training set and a test set in an 8:2 ratio. The training set of each sample pair included 826 sample pairs, and the test set included 206 sample pairs.

[0032] Reference Figure 2 The present invention provides a more detailed description of the construction and training of a deep supervised cross-modal retrieval network with a cross-attention mechanism.

[0033] Step 2: Construct a deep supervised cross-modal retrieval network with a cross-attention mechanism.

[0034] Step 2.1: Construct the 2D image feature extraction sub-network AlexNet, whose structure is as follows: First convolutional layer, first ReLU activation layer, first pooling layer, second convolutional layer, second ReLU activation layer, second pooling layer, third convolutional layer, third ReLU activation layer, fourth convolutional layer, fourth ReLU activation layer, fifth convolutional layer, fifth ReLU activation layer, third pooling layer, fourth pooling layer, first fully connected layer, sixth ReLU activation layer, first Dropout layer, second fully connected layer, seventh ReLU activation layer, second Dropout layer, third fully connected layer... The convolutional kernel sizes for the first to fifth convolutional layers are set to 11*11, 5*5, 3*3, 3*3, and 3*3, respectively, with a stride of 1 for each layer. The window size for the first to fourth pooling layers is set to 6*6, with a stride of 1 for each layer. The number of input and output nodes for the first to third fully connected layers are set to {9216,4096}, {4096,1024}, and {1024,2}, respectively. The ReLU activation function used for the first to seventh ReLU layers is max(x,0), and the dropout rate for the first to second Dropout layers is set to 0.5.

[0035] Step 2.2: Construct the one-dimensional parametric feature extraction subnetwork OmniCNN, whose structure is as follows: first convolutional layer, first batch normalization layer, second convolutional layer, second batch normalization layer, third convolutional layer, third batch normalization layer, first fully connected layer, Dropout layer, second fully connected layer; set the kernel size of the first to third convolutional layers to 7 and the stride to 1; set the number of input nodes and output nodes of the first and second fully connected layers to {64, 16} and {16, 2} respectively; set the numerical stability constant of the first to third batch normalization layers to 10. -5 The dropout rate for the Dropout layer is set to 0.5.

[0036] Step 2.3: Construct a cross-modal feature fusion sub-network. Its structure is a multi-head cross-attention mechanism layer. The number of attention heads in the cross-attention mechanism layer is set to 8, and the number of output nodes in the attention mechanism layer is set to 512.

[0037] Step 2.4: Construct a deep supervised learning subnetwork, whose structure is as follows: first pooling layer, first fully connected layer, and second fully connected layer; set the sliding window size of the pooling layer to 1*1 and the stride to 1; set the number of input nodes and output nodes of the first and second fully connected layers to {512,512} and {512,2}, respectively.

[0038] Step 2.5: The two-dimensional image feature extraction subnetwork and the one-dimensional parameter feature extraction subnetwork are connected in parallel, and then connected in series with the cross-modal feature fusion subnetwork and the deep supervised learning subnetwork to form a deep supervised cross-modal retrieval network with a cross-attention mechanism.

[0039] Step 3: Train a cross-modal retrieval network with deep supervision and cross-attention mechanism.

[0040] Step 3.1: Randomly select 32 previously unselected samples from the training set.

[0041] Step 3.2: Input the image data and parameter data from the selected sample pairs into the cross-modal retrieval network to obtain image features and parameter features containing salient information between them. Based on the extracted image features, parameter features, and class labels of the sample pairs, the discriminant loss value can be calculated using the following formula:

[0042]

[0043] Among them, L dist This represents the discriminative loss value between the image features and parameter features output by the cross-modal retrieval network training and their corresponding labels, where k represents the total number of sample pairs selected in the current iteration, ‖·‖ F This represents the Frobenius norm operation, where P represents the feature matrix output by the second fully connected layer of the deep supervised subnetwork for classification, and the superscript T indicates the transpose operation. This represents the image features output by the first fully connected layer of the deep supervised subnetwork in the cross-modal retrieval network. The label matrix representing the samples. This represents the parameter characteristics of the output of the first fully connected layer of the deep supervised subnetwork in the cross-modal retrieval network.

[0044] Step 3.3, based on the image features and parameter feature tags extracted by the cross-modal retrieval network, the similarity recognition loss value is calculated using the following formula:

[0045]

[0046]

[0047] Among them, L ident This represents the similarity recognition loss value between image features and parameter features output by the cross-modal retrieval network training output, where 1 · indicates that the value is 1 when two samples belong to the same class, and 0 otherwise. i Let u represent the image features in the i-th sample pair. j This represents the parameter features in the j-th sample pair. e i and uj The cosine distance between them.

[0048] Step 3.4, based on the image features and parameter feature tags extracted by the cross-modal retrieval network, the modality invariance loss value is calculated using the following formula:

[0049]

[0050] Among them, L model This represents the mode invariance loss value between image features and parameter features in the training output of the cross-modal retrieval network.

[0051] Step 3.5: In this invention, the three loss values ​​are weighted to obtain the overall training loss function, which can be expressed as:

[0052] L = L dist +λL ident +μL model

[0053] Where L represents the total loss between image features and parameter features output by the cross-modal retrieval network during training, and λ represents the constraint on L. ident The hyperparameters, λ, take a value of 0.001, and μ represents the constraint L. model The hyperparameter μ is set to 0.1.

[0054] Step 3.6: Iteratively update the network parameters using the backpropagation algorithm until the total loss value of the network converges, thus obtaining the trained cross-modal retrieval network.

[0055] Step 4: Retrieve aurora substorm images based on spatial environment parameters.

[0056] Step 4.1: Input the test set into the trained cross-modal retrieval network and output image features. and parameter features Calculate the feature similarity for each image and each pair of parameters. The feature similarity is obtained by the following formula:

[0057]

[0058] in, This represents the feature similarity matrix of the selected batch of sample pairs for images and parameters in the current iteration, where w represents the w-th sample pair and w represents the number of sample pairs.

[0059] Step 4.2: Based on the feature similarity obtained in Step 4.1, select the image with the highest similarity to each pair of parameters as the result of retrieving aurora substorm images.

[0060] The effects of this invention are further illustrated by the following simulation experiments.

[0061] 1. Simulation experimental conditions:

[0062] The hardware platform for the simulation experiment of this invention is: Intel(R) Core(TM) i5 CPU@2.6GHz, NVIDIA GeForce GTX TITAN X GPU, and 12G of memory.

[0063] The software platform for the simulation experiments of this invention is PyTorch.

[0064] The parameter dataset used in the simulation experiment of this invention consists of the three components of the interplanetary magnetic field IMF (Bx, By, Bz), solar wind velocity Vp, solar wind density Np, substorm index AE, and geomagnetic index SYM-H provided by NASA.

[0065] The aurora substorm image data used in the simulation experiment of this invention are ultraviolet aurora images acquired by the SSUSI imager on the DMSP satellite, and the data band is the Arctic LHBL band.

[0066] 2. Simulation content and result analysis:

[0067] The simulation experiment of this invention uses the method of this invention to retrieve aurora substorm images from the images in the test set using parameters with different combinations of parameters.

[0068] To evaluate the effectiveness of the simulation results of this invention, the evaluation criteria are as follows: Mean Average Precision (MAP), which takes into account both ranking information and query precision; and Recall Percentage at Top K (Rec@K), where K = 1, 5, 10. Rec@K represents the ratio of the similarity between each image and each set of parameters from largest to smallest, with the top K images of the highest similarity taken as the search results. The proportion of correct search results to all test data is statistically analyzed, and the results are shown in Table 1.

[0069] Table 1. Search results for different parameter combinations in this invention.

[0070]

[0071] Table 1 shows the simulation results obtained by combining different parameters as network inputs. This invention categorizes the seven space environment parameters closely related to substorms into four types: interplanetary magnetic field indices (Bx, By, Bz), solar wind indices (Vp, Np), auroral current indices (AE), and storm ring current indices (SYM-H). Although the Rec@5 and Rec@10 values ​​for selecting solar wind parameters (Vp, Np) and storm ring current indices (SYM-H) are slightly lower than those for selecting interplanetary magnetic field indices (Bx, By, Bz), solar wind indices (Vp, Np), and auroral current indices (AE), the average retrieval accuracy (MAP) is 7.62% higher. This demonstrates the best overall performance in retrieving auroral substorm images, indicating that this invention achieves both high retrieval accuracy.

Claims

1. A method for cross-modal aurora substorm image retrieval based on spatial environmental parameters, characterized in that, The image features and parameter features are cross-modal feature extraction and fusion by using attention mechanism and deep supervision learning, and the aurora substorm images are retrieved by using the spatial environment parameters; the retrieval method comprises the following steps: Step 1, generating a substorm event training set composed of spatial environment parameters and aurora images: At least 2000 ultraviolet aurora images of the LBHL wave band in the Arctic are selected, of which at least 1000 images contain substorm events; the spatial environment parameters corresponding to each ultraviolet aurora image are selected; The selected images and parameters are preprocessed respectively, each preprocessed image and its corresponding parameters form a sample pair, and each sample pair is labeled with a class label; all sample pairs and their corresponding class labels form a training set; The preprocessing is to crop each image to 224x224 size with (70MLAT, 21MLT) coordinates as the center, and divide each cropped image into 36 region blocks to obtain the preprocessed image data; the spatial parameters corresponding to each image are grouped, and the interplanetary magnetic field and solar wind parameters are delayed by 14 minutes to obtain the preprocessed parameter data; Step 2, constructing a deep supervision cross-modal retrieval network based on cross-attention mechanism, which is composed of a two-dimensional image feature extraction subnetwork and a one-dimensional parameter feature extraction subnetwork in parallel, and then a cross-modal feature fusion subnetwork and a deep supervision learning subnetwork in series; Step 3, training the cross-modal retrieval network with cross-attention mechanism deep supervision: The training set is input into the cross-modal retrieval network in batches, the discriminant loss value between the output of the network and the class label of the sample is calculated, the discriminant loss value, the similarity recognition loss value of the image features and the parameter features, and the modal invariance loss value are weighted and summed to obtain the total loss value of the cross-modal retrieval network, the parameters of the network are updated by the back propagation algorithm, and the total loss value of the network is converged until the total loss value of the network is converged, and the trained cross-modal retrieval network is obtained; Step 4, retrieving aurora substorm images by using spatial environment parameters: The same method as step 1 is used to preprocess the images to be retrieved and the corresponding spatial environment parameters, and the preprocessed images to be retrieved and the parameters are input into the trained cross-modal retrieval network, the feature similarity of each image and each pair of parameters is calculated, and the image with the highest similarity for each pair of parameters is selected as the result of retrieving aurora substorm images.

2. The aurora substorm image retrieval method across modalities based on spatial environment parameters according to claim 1, characterized in that, The structure of the two-dimensional image feature extraction subnetwork described in step 2 is in turn: a first convolutional layer, a first ReLU activation layer, a first pooling layer, a second convolutional layer, a second ReLU activation layer, a second pooling layer, a third convolutional layer, a third ReLU activation layer, a fourth convolutional layer, a fourth ReLU activation layer, a fifth convolutional layer, a fifth ReLU activation layer, a third pooling layer, a fourth pooling layer, a first fully connected layer, a sixth ReLU activation layer, a first Dropout layer, a second fully connected layer, a seventh ReLU activation layer, a second Dropout layer, and a third fully connected layer. The convolution kernel sizes of the first to fifth convolutional layers are set to 11*11, 5*5, 3*3, 3*3, and 3*3, respectively, and the step size is set to 1. The window sizes of the first to fourth pooling layers are all set to 6*6, and the step size is set to 1. The input nodes and output nodes of the first to third fully connected layers are set to {9216, 4096}, {4096, 1024}, and {1024, 2}, respectively. The first to seventh ReLU activation layers use the function max(x, 0), and the dropout rates of the first to second Dropout layers are both set to 0.

5. 3.The aurora substorm image retrieval method based on spatial environment parameter cross-modality according to claim 1, characterized in that, The structure of the one-dimensional parameter feature extraction subnetwork described in step 2 is in turn: a first convolutional layer, a first batch normalization layer, a second convolutional layer, a second batch normalization layer, a third convolutional layer, a third batch normalization layer, a first full connection layer, a Dropout layer, and a second full connection layer; the convolution kernel size of the first to third convolutional layers is set to 7, and the step is set to 1; the number of input nodes and output nodes of the first and second full connection layers is set to {64, 16} and {16, 2} respectively, and the numerical stability small constant of the first to third batch normalization layers is set to 10 -5 , and the dropout rate of the Dropout layer is set to 0.

5.

4. The method of claim 1, wherein the method is based on spatial environmental parameters across modal aurora substorm image retrieval. The structure of the cross-modal feature fusion subnetwork described in step 2 is a multi-head cross-attention mechanism layer, and the number of attention heads of the cross-attention mechanism layer is set to 8, and the output node number of the attention mechanism layer is set to 512.

5. The method of claim 1, wherein the method is based on spatial environmental parameters across modal aurora substorm image retrieval. The structure of the deep supervised learning subnetwork described in step 2 is in turn: a first pooling layer, a first fully connected layer, and a second fully connected layer. The sliding window size of the pooling layer is set to 1*1, and the step size is set to 1. The input nodes and output nodes of the first and second fully connected layers are set to {512, 512} and {512, 2}, respectively.

6. The cross-modal aurora substorm image retrieval method based on spatial environment parameters according to claim 1, characterized in that, The discriminative loss value described in step 3 is obtained by the following formula: wherein, L dist represents the discriminative loss value of the image feature and the parameter feature output by the cross-modal retrieval network training and its corresponding label, k represents the total number of selected sample pairs in the batch at the current iteration, ‖·‖ F represents the Frobenius norm operation, P represents the feature matrix output by the second full connection layer of the deep supervision subnetwork, the upper index T represents the transposition operation, represents the image feature output by the first full connection layer of the deep supervision subnetwork in the cross-modal retrieval network, represents the label matrix of the sample, represents the parameter feature output by the first full connection layer of the deep supervision subnetwork in the cross-modal retrieval network.

7. The cross-modal aurora substorm image retrieval method based on spatial environment parameters according to claim 6, characterized in that, The similarity recognition loss value described in step 3 is obtained by the following formula: wherein, L ident represents the similarity recognition loss value between the image features and the parameter features of the cross-modal retrieval network training output, 1{·} represents that when two samples belong to the same class, its value is 1, and otherwise 0, e i represents the image feature of the i-th sample pair, u j represents the parameter feature of the j-th sample pair, represents the cosine distance between e i and u j .

8. The cross-modal aurora substorm image retrieval method based on spatial environment parameters according to claim 7, characterized in that, The modality invariance loss value described in step 3 is obtained by the following formula: wherein L model represents the modal invariance loss value between the image features and the parameter features output by the cross-modal retrieval network training.

9. The cross-modal aurora substorm image retrieval method based on spatial environment parameters according to claim 8, characterized in that, The total loss value of the cross-modal retrieval network described in step 3 is obtained by the following formula: L = L dist + λL ident + μL model Wherein, L represents the total loss value between the image feature and the parameter feature of the cross-modal retrieval network training output, λ represents the hyperparameter of limiting L ident , λ is 0.001, μ represents the hyperparameter of limiting L model , μ is 0.

1.

10. The cross-modal aurora substorm image retrieval method based on spatial environment parameters according to claim 9, characterized in that, The feature similarity described in step 4 is obtained by the following formula: wherein, represents the feature similarity matrix of the selected batch of samples to the image and parameters at the current iteration, w represents the wth sample pair, and w represents the number of sample pairs.

Images