Multi-modal time-series astronomical data training method and device based on graph contrastive learning and storage medium

Abstract

The application provides a multi-modal time sequence astronomical data training method and device based on graph contrast learning and a storage medium, and belongs to the field of astronomical data processing; solves the limitations of existing deep learning methods in processing multi-modal astronomical data fusion; the method includes three key steps: an astronomical data mapping model is responsible for structuring the original data; a multi-modal astronomical time sequence data fusion model based on a self-attention mechanism can effectively integrate data from different observation periods and different modalities, and realize deeper feature fusion; a feature learning model based on graph contrast learning further extracts more discriminative and representative features at the graph structure level; the application is aimed at multi-modal data of astronomical image data, spectral data, star table data and text data, and combines data of different time dimensions for feature fusion, and through the mapping-fusion-re-mapping mode, more specific features are mined from the graph perspective by using graph contrast learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of astronomical data processing, and particularly relates to a multi-modal time-series astronomical data training method and device based on graph contrastive learning and a storage medium. BACKGROUND

[0002] In the field of astronomical research, with the continuous development of observation technology, astronomers can obtain multi-modal observation data about the same celestial body through different waveband telescopes (such as optical, radio, X-ray, etc.) and spectrographs and other equipment. These multi-modal data contain information about the celestial body in different physical dimensions, such as spatial morphology, spectral characteristics, time evolution, etc. At the same time, many important astronomical phenomena (such as supernova explosions, exoplanet transits, etc.) have significant time dependence, so processing and analyzing multi-modal time-series astronomical data is crucial for understanding celestial physical processes, discovering new celestial bodies and new phenomena.

[0003] However, processing multi-modal time-series astronomical data faces many challenges. First, the data of different modalities differ significantly in physical properties, morphological distribution, structural characteristics, etc., and how to align these different data is a problem that needs to be considered. Second, time-series data itself contains dynamic evolution information, and the observation times of different modalities may be completely out of sync, so how to effectively use this time-series information is also a difficult problem. At the same time, the amount of data generated by astronomical observation is extremely large, and it is necessary to improve the efficiency and scalability of data processing.

[0004] To address these challenges, researchers have tried various methods. Most processing methods are through traditional statistical methods or rule-based methods, which often have difficulty processing more complex and high-dimensional data. In recent years, machine learning and deep learning technologies have been widely applied in astronomical data analysis due to their powerful data processing and feature extraction capabilities. For example, convolutional neural networks (CNN) are often used for image processing, and recurrent neural networks (RNN) and their variants (such as LSTM, GRU) are often used for processing time-series data. Some researches also try to simply combine or stack these models to process multi-modal time-series data.

[0005] However, existing deep learning methods still have limitations in processing multi-modal astronomical data fusion. On the one hand, traditional fusion strategies (such as feature-level concatenation) may not be able to fully exploit the complementary information and deep correlations between different modal data, resulting in poor fusion results. On the other hand, in feature extraction, although deep learning models have the ability to automatically learn features, they may be susceptible to data noise, observation bias, or the extracted features may be biased towards shallow patterns in the data, failing to fully capture complex, deep astrophysical features resulting from inter-modal interactions. Therefore, how to use self-supervised methods such as contrastive learning to learn more discriminative and robust representations from unlabeled multi-modal time series data still needs further research. SUMMARY

[0006] To solve the above technical problems, the present application proposes a multi-modal time series astronomical data training method based on graph contrastive learning, device and storage medium.

[0007] The technical solution adopted by the present application is: a multi-modal time series astronomical data training method based on graph contrastive learning, comprising the following steps:

[0008] I. Data preparation and preprocessing: collect and preprocess astronomical data of multiple modalities to obtain multi-modal astronomical data that are correlated in time series;

[0009] II. Constructing a multi-modal astronomical time series data graph model: abstracting the basic units in each modality and each time series astronomical data as nodes in the graph, constructing edges by calculating the similarity between the features carried by the nodes, and obtaining a series of graph sets reflecting different times and different modalities;

[0010] III. Constructing a multi-modal astronomical time series data fusion model based on self-attention mechanism, inputting the graph structure corresponding to each time step and each modality into the graph neural network to extract node-level feature representation, capturing the node's own attributes and its interaction information with neighboring nodes, then performing time series feature fusion on the multi-modal astronomical data with time series attributes through the multi-modal astronomical time series data fusion model, and performing cross-modal feature concatenation and self-attention fusion on the extracted multi-modal features of each node to obtain a unified feature representation that has fused all modal information;

[0011] IV. Constructing a feature learning model based on graph contrastive learning: constructing a new graph from the unified feature representation, and constructing a saliency score graph based on the new graph, obtaining an enhanced graph through an enhancement strategy based on the saliency score graph, then selecting appropriate positive and negative sample pairs for comparison, calculating the contrastive loss and performing feature optimization and model training to obtain an updated unified feature representation.

[0012] Further, the multi-modal astronomical data at least includes: image data, spectral data, star catalog data and text data. After collecting the above multi-modal astronomical data in step one, data alignment and time division are performed first. Astronomical image data and spectral data are divided into multiple time windows or time points according to observation time stamps.

[0013] The four kinds of multi-modal data are respectively preprocessed to obtain image patches, continuous wavebands of spectral data, records of star catalogs and embedding vectors of text fragments or key concept words, and the embedding vectors are taken as basic units of the respective modal data.

[0014] Further, the multi-modal astronomical time series data graph construction model in step two is constructed as follows:

[0015] S2.1: The basic units in the respective modal data are abstracted as nodes in a graph, and the connection relationship between the nodes, i.e., the edges, is constructed by calculating the similarity between the features carried by the nodes, thereby generating a corresponding graph structure for the data of each modality;

[0016] S2.2: For the image data and spectral data with time series attributes, the data is further graphed according to the time sequence. For the data at each time point, the above graphing process is repeated;

[0017] S2.3: A series of time evolution graphs reflecting the data state at different times are generated, thereby obtaining a dynamic and multi-modal graph set.

[0018] Further, the specific process of time series feature fusion and cross-modal feature fusion by the multi-modal astronomical time series data fusion model in step three is as follows:

[0019] S3.1: For the image data and spectral data with time series attributes, an average pooling strategy is adopted to integrate the time series information, and the time series information is compressed into a fixed-dimensional feature vector, thereby generating a unified preliminary feature representation for the image data and the spectral data;

[0020] S3.2: The processed preliminary features of the four modalities are spliced along the node feature dimension, and the spliced feature vector is input into a self-attention mechanism, and finally a unified feature representation that integrates the information of the four modalities and is more discriminative is output.

[0021] Further, the process of constructing the unified feature representation into a new graph in step four is as follows:

[0022] Each feature vector in the unified feature representation is regarded as a node in the graph G, and the edges of the graph are constructed by calculating the similarity between these node features, thereby forming a new graph structure reflecting the relationship between the features as an input graph in graph contrast learning.

[0023] Further, the process of constructing the saliency score graph based on the new graph in step four is as follows:

[0024] First, the initial saliency score of each node in the new graph is calculated by a gradient-based method;

[0025] Then, the initial saliency score of each node is optimized by aggregating the saliency scores of the current node and its adjacent nodes, so as to obtain the final node saliency s v ;

[0026] For edge saliency, it is obtained by taking the average of the saliency scores of the two nodes connected by the edge.

[0027] Further, the enhancement strategy based on the saliency score graph in step four includes two parts: node feature masking and edge weight change;

[0028] The strategy of node feature masking is to retain the node features that have a greater impact on the saliency score graph based on the guidance of the node saliency score, and randomly mask those node features that have a smaller impact;

[0029] The strategy of edge weight change is to replace the original weight of the corresponding edge in the adjacency matrix with the saliency score of the edge, i.e. 1 in the original adjacency matrix.

[0030] Further, the process of selecting and calculating the contrast loss of positive and negative sample pairs in step four is as follows:

[0031] Positive sample pair: the embedding representation of the same node in different enhanced views and the embedding representation of the neighbor nodes of the anchor node within the same view and between different views are regarded as a positive sample pair;

[0032] Negative sample pair: the embedding representation of the non-neighbor nodes of the anchor node within the same view and the embedding representation of the non-neighbor nodes of the anchor node between different views are regarded as a negative sample pair;

[0033] The contrast loss is designed by pulling the distance between the positive sample pairs closer and pushing the distance between the negative sample pairs farther.

[0034] A computer device comprising a memory, a processor and a computer program stored on the memory, the processor executing the computer program to implement the steps of the method.

[0035] A computer readable storage medium having stored thereon a computer program / instructions, which, when executed by a processor, implements the steps of the method.

[0036] The application has the beneficial effects relative to the prior art: the application learns the characteristics of multi-modal time series data from a different angle than before. It is proposed to model the complex and potential correlation between data points and internal elements of data by means of graph structure, and at the same time, to learn more representative and higher-discriminative feature representations from multi-modal time series astronomical data by combining contrast learning mechanism, thereby significantly improving the accuracy and efficiency of astronomical data analysis.

[0037] The application is aimed at multi-modal data of astronomical image data, spectral data, star catalog data and text data, and combines feature fusion of data with different time dimensions. By means of graph- fusion-regraph, more specific features are mined from the perspective of graph by using graph contrast learning. The method has good generalization, and the introduction of graph contrast learning model guarantees the robustness and stability of feature extraction. Astro-GTS provides an end-to-end feature extraction solution, opening up a new way for intelligent analysis and understanding of complex astronomical data. BRIEF DESCRIPTION OF DRAWINGS

[0038] The application will be further described below in conjunction with the drawings:

[0039] Figure 1 Astro-GTS framework provided for the embodiments of the application;

[0040] Figure 2 Data fusion structure diagram based on self-attention mechanism provided for the embodiments of the application;

[0041] Figure 3 Significance score diagram acquisition process provided for the embodiments of the application. DETAILED DESCRIPTION

[0042] As shown in Figures 1 to 3 The application provides a multi-modal time series astronomical data training method based on graph contrast learning. A new network structure, namely Astro-GTS network, is proposed. The Astro-GTS network is composed of a multi-modal astronomical time series data graphing model, a multi-modal astronomical time series data fusion model based on a self-attention mechanism, and a feature learning model based on graph contrast learning.

[0043] The multi-modal astronomical time series data composition model is used to abstract the basic units in each mode and each time series data as nodes in a graph, build edges by calculating the similarity between the features carried by the nodes, and then obtain a series of graph sets reflecting different times and different modes. The multi-modal astronomical time series data fusion model based on the self-attention mechanism includes two parts: feature fusion for time series attribute data (astronomical image data and spectral data), and multi-modal feature fusion based on the self-attention mechanism. The feature learning model based on graph contrast learning includes three parts: multi-modal time series data fusion feature composition, saliency score graph construction, and graph enhancement based on the saliency score graph.

[0044] The method for realizing data training of the present application mainly includes the following steps:

[0045] I. Data preparation and preprocessing, specifically including:

[0046] 1. First, collect astronomical data of four modes:

[0047] (1) Astronomical image data: can be obtained through the official data access platform of Sloan Digital Sky Survey (SDSS), SkyServer platform. On this platform, users can query and download image data of stars and other celestial bodies taken at different times. This embodiment focuses on star targets, but SDSS data is provided in FITS format, which needs to be converted to image format.

[0048] (2) Spectral data: also obtained from the SDSS SkyServer platform, corresponding to the image target described above. Ensure that the spectral data matches the image data in terms of target celestial body and observation time.

[0049] (3) Star catalog data: collect star catalog data containing the basic attributes (such as coordinates, apparent magnitude, proper motion, spectral type, etc.) of target celestial bodies (stars). These data come from SDSS DR (Data Release). Ensure that the star catalog data is consistent with the target of the image and spectral data.

[0050] (4) Text data: collect scientific literature abstracts, observation reports, classification labels, and other text information related to target celestial bodies. Can be obtained through astronomical databases (such as NASA / IPAC Infrared Astronomical Physics Database, ADS Literature Library) or manual retrieval. The text data needs to be associated with the data obtained before.

[0051] 2. Data alignment and time division: used to ensure that different modal data are aligned in time (for example, data of the same observation target at similar time points).

[0052] For image and spectral data, divide them into multiple time windows or time points (t1, t2, …, tq Each time window contains the image and spectrum data at that moment. The star catalog and text data are usually not strictly time-ordered and are divided into time sequences in disorder.

[0053] 3. Data preprocessing:

[0054] (1) Astronomical image data: Check image quality, remove low-quality or invalid images caused by weather, instrument failure, etc. Normalize pixel values, perform necessary denoising, alignment, etc. Divide the image into pixels or local regions (such as 3x3 or 5x5 patches) as node candidates.

[0055] (2) Spectrum: Check spectrum quality, remove spectra with unstable baseline, too low signal-to-noise ratio, or obvious instrument failure signs. Normalize light intensity values, divide the spectrum into consecutive waveband intervals as node candidates.

[0056] (3) Star catalog: Clean up entries with too many missing values or obvious errors in the star catalog. Unify the coordinate system to ensure consistency with the coordinates of images and spectra. Select key features (such as coordinates, brightness, and magnitude), and each record is a node.

[0057] (4) Text: Remove irrelevant or duplicate text information. Perform word segmentation, remove stop words, and use word embeddings (such as Word2Vec) or sentence encoders (such as BERT) to convert text fragments / key concepts into vector representations, with each vector representing a node.

[0058] (5) Unified time index: Assign a unified time index to all data for subsequent processing.

[0059] To effectively fuse four key astronomical observation modalities - astronomical image data, spectrum data, star catalog data, and text data, the present application proposes a multi-modal astronomical time series data graph modeling model, the core idea of which is to use graph structure to bridge the differences between different modalities, realize data alignment, and lay the foundation for subsequent feature extraction.

[0060] Specifically, first, the basic units in each modality data are abstracted as nodes in the graph: in image data, nodes represent pixels or local regions; in spectrum data, nodes correspond to specific spectral bands; in star catalog data, nodes are single records in the star catalog; and in text data, nodes can be regarded as text fragments or key concepts. Subsequently, by calculating the similarity between the features carried by these nodes, the connection between nodes, i.e., edges, is established. In this way, a corresponding graph structure is generated for the data of each modality, and these graph structures can capture the internal relationships of the modality data.

[0061] For data with temporal attributes (such as images and spectra), the observation data is further divided by time series. For the data at each time point, the above graph construction process is repeated, thereby generating a series of time evolution graphs reflecting the state of data at different times. In this way, a dynamic and multi-modal graph set is obtained.

[0062] II. Multi-modal astronomical time series data graph construction model, specifically comprising:

[0063] 1. Node definition, including the following four kinds of nodes:

[0064] Image: the pre-processed image patch (such as 5x5) is regarded as a node;

[0065] Spectrum: the continuous wave band (such as 10nm width) of the spectrum is regarded as a node;

[0066] Star Catalog: each record of the star catalog is regarded as a node;

[0067] Text: the embedding vector of a text segment (such as a sentence) or a key concept word is regarded as a node;

[0068] 2. Edge construction:

[0069] For each time point t q and each modality m k (representing image, spectrum, star catalog or text):

[0070] The similarity between the features carried by the nodes is calculated using cosine similarity to construct edges. A similarity threshold is set, and the k-neighbor method is used to connect the node pairs with similarity exceeding the threshold to form edges.

[0071] 3. Generate the corresponding graph structure under different time series and different modalities:

[0072] Generate the graph G k corresponding to time point t q in modality m k (q) =(V k (q) , E k (q) ), where V k (q) is the node set, and E k (q) is the edge set. For star catalog and text data, time series is not considered, and only the graph structure under the corresponding modality is generated.

[0073] Finally, a dynamic and multi-modal graph set is obtained. For astronomical image data, G1 (q) is obtained, and q represents the number of time series; for spectral data, G2 (q); G3 for the catalog data; G4 for the text data.

[0074] After successfully constructing the set of time-ordered multi-modal graphs, we enter the feature fusion stage. First, we input the graph structure corresponding to each time step and each modality into the graph neural network (GNN). The GNN is responsible for extracting node-level feature representations from these complex graph structures, capturing the node's own attributes and its interaction information with neighboring nodes.

[0075] Next, we perform feature fusion in two steps. 1. For data modalities with time-series properties, i.e., astronomical images and spectral data, we use an average pooling strategy to integrate their time-series information. Specifically, for the feature sequence of each node extracted by GNN at all time steps, we calculate the average value of this sequence in the time dimension. In this way, we compress the time-series information into a fixed-dimensional feature vector, thereby generating a unified feature representation for the image and spectral modalities. 2. We concatenate the four modality features (image, spectrum, catalog, and text) for each node to form a longer node feature vector. Subsequently, we input all the concatenated node feature vectors into the self-attention mechanism.

[0076] III. Multi-modal astronomical time-series data fusion model based on self-attention mechanism

[0077] 1. GNN feature extraction:

[0078] For each graph G in the graph set k (q) , use the GNN model for feature extraction, input the node features and graph structure (adjacency matrix) into the GNN. Through multi-layer propagation, update the feature representation of each node. Output the features of the nodes in the graph structure at time point t q .

[0079] For example Figure 1 , there are four modalities, and each modality has eight nodes. For the time series in modality , the feature of node can be represented as , where k represents the modality , and q represents the time series . The same applies to other time series, and the node features corresponding to different modalities at different time series are obtained. Then, the average of all node features in the time series dimension is calculated to obtain the fused node features .

[0080] 2. Time-series feature fusion (image and spectrum):

[0081] ​​For image and spectral data, it is necessary to integrate them at multiple time points t1, t2, …, t q The extracted features are fused to obtain a comprehensive representation of the data in each modality in the time dimension.

[0082] Specifically, average pooling is used in the time dimension for each node feature to obtain a unified node feature of the fused time series, in this way, the fused features of the astronomical image time series data and the fused features of the spectral time series .

[0083] 3. Cross-modal feature concatenation and self-attention fusion:

[0084] After obtaining the preliminary feature vectors representing four different modalities of information , , , , these feature vectors are concatenated along the node feature dimension, that is, the node features of the four modalities are concatenated for each node. For example, for node , the concatenated feature vector is . Then, the feature vectors of all concatenated nodes are input into the self-attention layer for further feature fusion.

[0085] The self-attention layer calculates Q (query matrix), K (key matrix), and V (value matrix), and through Q and K, it calculates the mutual attention between the internal different modal feature vectors, that is, the attention matrix. These weights reflect the relative importance of different modal information in the current context. Based on these calculated weights, the weighted sum of each part of the concatenated vector is performed. Finally, the output is a unified feature representation that integrates all modal information, which can more comprehensively capture the complex relationships between multi-modal data.

[0086] After successfully fusing the four modalities of information and obtaining the unified feature representation H, a graph contrast learning mechanism is introduced to perform deeper feature extraction on H from the perspective of graph structure.

[0087] Four, feature learning model based on graph contrast learning

[0088] 1. Graph construction based on unified feature representation H:

[0089] Each node feature in the fused unified feature representation H The node features in the new graph G are regarded as the node features in the new graph G. The similarity (such as cosine similarity) between these node features is calculated to construct the edges of the graph, forming the graph G=(X, A), where X is the feature matrix of the graph, i.e. H, and A is the adjacency matrix of the graph.

[0090] 2. Significant score graph construction:

[0091] In order to identify which nodes and edges in the graph are more critical to the learning process, a significant score graph is used to make the judgment. After constructing the graph G, it is used as the input graph for graph contrastive learning to train. First, based on the graph G, a significant score graph G score is calculated. This graph assigns a significant score s v ∈[0, 1] to each node and a significant score s e ∈[0, 1] to each edge to quantify their importance to the entire graph. The larger these significant scores, the greater the influence of the node or edge in the graph. These significant scores will serve as a guide for subsequent graph enhancement operations.

[0092] The construction of the significant score graph is a continuous iterative updating process. In the initial stage, since the significant score graph has not been generated to guide view enhancement, the system will use random enhancement as a starting point for loss calculation. As learning progresses, after each loss calculation, the model will obtain the gradient of the node feature on the loss through backpropagation, and use these gradients to update the significant score graph. In this way, the significant score graph can continuously adapt to the results of model learning, ensuring that the enhanced graph generated can more accurately preserve important semantic information in the graph.

[0093] The specific process is as follows: First, the initial significant score of each node is calculated based on the gradient-based method. Then, the initial result is optimized by aggregating the significant scores of the current node and its adjacent nodes, to obtain the final node significant score s v which aggregates the significant scores of the node itself and its neighbor nodes. For edge significance, it is obtained by taking the average of the significant scores of the two nodes connected. Figure 3 The complete process of generating a significant score graph is intuitively shown, mainly including three key steps: (1) single node significant score calculation; (2) significant fusion of itself and neighbor nodes; (3) significant score sorting.

[0094] (1) Single node significant score calculation:

[0095] First, the forward propagation of the graph neural network (GNN) is used to obtain the initial feature embedding representation of each node . Then, a gradient-based method is used to evaluate the importance of a single node by calculating the contrastive loss Regarding node features The gradient is obtained with respect to the node features. The contrastive loss is obtained with two augmented graphs and corresponding embeddings The metric reflects the influence degree of individual nodes in the graph. Where and are different augmented graphs obtained by setting different parameter ratios to G based on the saliency score. Since the saliency score graph cannot be obtained in the initial augmentation stage, a random augmentation is used for the initial state when calculating the gradient of the loss with respect to the node features. The formula for calculating node saliency is: where represents the influence degree of the feature of node on the graph, which quantifies the rate of change in loss between when and occurs a slight change, and the greater the value, the greater the influence of the node in the graph. represents the embedding corresponding to the augmented graph and .

[0096] (2) Saliency fusion with neighbor nodes:

[0097] In graph neural networks, the saliency of a single node may not fully reflect its actual role in the graph, especially in complex network structures, where the meaning of a node is often closely related to the characteristics of its neighbor nodes. Therefore, through the method of saliency fusion, by considering the saliency of the node and its neighbor nodes, the overall importance of the node is more accurately evaluated.

[0098] The specific node saliency fusion score is obtained by the following formula: . Where, represents the saliency fusion score of node , is the original saliency score of node , represents the neighbor node set of node , is the weight coefficient, used to balance the contribution of the node's own saliency and neighbor node saliency to the final score.

[0099] After obtaining the node saliency fusion score, the edge saliency score is the average of the connected node saliency scores: where and are the saliency fusion scores of node and node , the significance score of the edge.

[0100] (3) Significance ranking:

[0101] The significance scores of the obtained nodes and edges are ranked, such as Figure 3 The darker the color of a node indicates the greater the significance of the node, and similarly, the thicker the edge indicates the greater the significance of the edge. The guidance based on the size of the significance score provides a basis for subsequent enhancement operations.

[0102] 3, the graph enhancement strategy based on the significance score graph, the traditional random graph enhancement strategy often masks nodes or edges randomly to increase data diversity, but this may cause the key nodes / edges that contribute more to the graph to be accidentally removed, which not only weakens the feature learning effect, but also may destroy the topology of the graph, so that the learned feature information is insufficient. To overcome this problem, the present application designs an enhancement strategy based on a significance score graph, which aims to preferentially retain the graph topology and key information while still increasing the diversity of the enhanced view. The strategy mainly includes two parts: node feature masking and edge weight changing.

[0103] For the enhancement operation of node feature masking, the main idea is to guide based on the node significance score, and the node features that have a greater impact on the significance score graph (large significance score value) are retained, and only the node features that have a smaller impact are randomly masked.

[0104] For the enhancement operation of edge weight changing, the main idea is to reflect the importance of the edge (i.e. the significance score) in the adjacency matrix, so that the significance of the edge can be considered in the subsequent feature aggregation.

[0105] Specifically as follows:

[0106] (1) Node enhancement: According to the significance score of the nodes in the significance score graph, the importance of the nodes is ranked. When generating the enhanced view, the node features with high significance are preferentially retained, and only the node features with low significance are masked. This ensures that the information of the nodes with large contribution is retained, while also increasing the diversity of the enhanced view.

[0107] Specifically, the node subset that can be masked features is determined according to the masking probability of each node. The masking probability is obtained based on the node significance fusion score . Using the Bernoulli distribution obtained, represents the possibility that the features of node can be masked. is a hyperparameter that is used to adjust the degree of influence of significance. By setting different and Two different augmented graphs can be generated and .

[0108] (2) Edge augmentation: In order to avoid destroying the topology of the graph by random augmentation, no edge is deleted. Instead, the significance score of the edge is used to adjust its importance, that is, the original weight 1 of the corresponding edge in the adjacency matrix is replaced by the significance score of the edge. In this way, the topology of the graph is maintained, and the importance information of the edge is also integrated, and the aggregation of the node features in the subsequent process can be based on the significance of the edge.

[0109] The augmented view generated by this strategy not only protects the topology and important semantic information of the graph, but also increases the diversity of the data, laying a good foundation for subsequent feature learning.

[0110] The significance score of the edge is used to redefine the weight of the edge. By quantifying the contribution of the edge, the essential features of the graph can be more accurately captured. Specifically, traditional graph processing methods often treat edges as homogeneous elements, ignoring their differences in information aggregation. In this application, it is believed that some edges may have a more important role in the graph structure and have a decisive influence on the properties and functions of the graph. Therefore, directly converting the significance score of these edges into edge weights can emphasize their importance in the graph structure. Through this method, a new adjacency matrix whose elements are determined by . For edges that are connected in the original graph, the significance score of the edge is used to replace the 1 in the original adjacency matrix, while for edges that are not connected, they are still not connected to achieve the purpose of preserving the original topology.

[0111] 4. Contrastive loss calculation and optimization, by setting different parameters for the significance ratio (i.e. ), two views augmented by the significance score graph are generated, and they are respectively input into the encoder (Encoder) and the projection head (ProjectionHead). For each view, the embedding representation of the node is calculated. Subsequently, the contrastive loss is calculated by pulling the distance between positive sample pairs and pushing the distance between negative sample pairs.

[0112] Specifically, it includes:

[0113] (1) Encoding and projection: two augmented views and are respectively input into the GNN encoder to learn the embedding of each node by aggregating the embedding of the neighbors through the adaptive edge coefficient, and then input into the projection head Projection Head to obtain the final contrastive embedding and .

[0114] (2) Positive and negative sample pair definition:

[0115] Positive sample pair: The embedding representation of the same node in different augmented views and the embedding representation of the neighbor nodes of the anchor node within the same view and between different views are regarded as a positive sample pair.

[0116] Negative sample pair: The embedding representation of the non-neighbor nodes of the anchor node within the same view and the embedding representation of the non-neighbor nodes of the anchor node between different views are regarded as a negative sample pair.

[0117] (3) Loss calculation and optimization: Let and represent the L2 normalized embedding of node in two augmented views, and let be the anchor point, then its positive samples mainly include the embedding of the same node in different views , the embedding of the neighbor node in the same view , and the embedding of the neighbor node in different views . Then the corresponding neighbor contrastive loss is represented as:

[0118] ;

[0119] In the formula: is the number of positive samples, is the temperature parameter, represents the similarity measure (inner product). Since the two views are symmetric, given the embedding Figure 2 of node in view as the anchor point, the neighbor contrastive loss can also be calculated by the above formula. Therefore, the average value of the final neighbor contrastive loss of view Figure 1 and view Figure 2 on all nodes is defined as:

[0120] .

[0121] By optimizing the loss function by gradient, the optimal encoder parameters can be trained. These parameters can learn more intrinsic correlation features in the data, and finally be used to update the unified feature representation H of the astronomical time series fusion data.

[0122] The computer device includes a memory and a processor, and the memory stores instructions executable on the processor. The processor implements the method in the above embodiments when executing the instructions. The number of memories and processors can be one or more. The computer device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The computer device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit the present application described and / or claimed in this document to that which is presented herein.

[0123] The computer device can also include a communication interface to facilitate communications with other devices. The various devices are interconnected by various buses, and can be mounted on a common motherboard or in other manners as appropriate. The processor can process instructions for execution within the computer device, including instructions stored in the memory or on the storage device to display graphical information for a GUI on an external input / output device, such as a display device coupled to the interface. In other implementations, multiple processors and / or multiple buses can be employed as appropriate, as well as multiple memories and types of memory. Also, various

[0124] Optionally, if the memory, the processor and the communication interface are integrated on a chip, the memory, the processor and the communication interface can communicate with each other through an internal interface.

[0125] It should be appreciated that the above processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. It should be noted that the processor can be a processor supporting an advanced RISC machine (ARM) architecture.

[0126] The embodiment of the present application provides a computer readable storage medium (such as the memory described above), which stores computer instructions, and the program is executed by a processor to implement the method provided in the embodiment of the present application.

[0127] Optionally, the memory can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; and the data storage area can store data created according to the use of the mapping computer device, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-transient memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transient solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the mapping computer device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0128] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the above embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training multimodal time-series astronomical data based on graph contrastive learning, characterized in that: Includes the following steps: Step 1: Data preparation and preprocessing: Collect astronomical data of multiple modes and preprocess them to obtain multimodal astronomical data that are temporally correlated; Step 2: Construct a multimodal astronomical time-series data graph model: Abstract the basic units in each modality and time-series astronomical data into nodes in the graph. By calculating the similarity between the features carried by the nodes, construct edges to obtain a series of graph sets reflecting different times and modalities. Step 3: Construct a multimodal astronomical time-series data fusion model based on self-attention mechanism. Input the graph structure corresponding to each time step and each modality into the graph neural network to extract node-level feature representations, capture the node's own attributes and its interaction information with neighboring nodes, and then use the multimodal astronomical time-series data fusion model to fuse the time-series features of multimodal astronomical data with time-series attributes. Then, perform cross-modal feature concatenation and self-attention fusion on the multimodal features of each extracted node to obtain a unified feature representation that integrates all modal information. The specific process of performing time series feature fusion and cross-modal feature fusion using the multimodal astronomical time series data fusion model in step three is as follows: S3.1: For image data and spectral data with temporal attributes, an average pooling strategy is used to integrate their temporal information, compressing the temporal information into a fixed-dimensional feature vector, and generating a unified preliminary feature representation for both image data and spectral data. S3.2: The preliminary features of the four modalities after processing are concatenated along the node feature dimension. The concatenated feature vector is then input into the self-attention mechanism, and finally a unified feature representation that integrates the information of the four modalities and is more discriminative is output. Step 4: Construct a feature learning model based on graph contrastive learning: Construct a new graph from the unified feature representation, and construct a saliency score graph based on the new graph. Obtain an augmented graph through an augmentation strategy based on the saliency score graph. Then, select positive and negative sample pairs for comparison, calculate the contrastive loss, and perform feature optimization and model training to obtain the updated unified feature representation. The process of constructing the significance score plot based on the new plot in step four is as follows: First, the initial saliency score of each node in the new graph is calculated using a gradient-based method; Next, by aggregating the saliency scores of the current node and its neighboring nodes, the initial saliency score of each node is optimized, thereby obtaining the final node saliency that aggregates the node's own saliency and the saliency of its neighboring nodes. v ; For edge saliency, it is obtained by averaging the saliency scores of the two nodes it connects; The augmentation strategy based on the saliency score graph in step four consists of two parts: node feature masking and edge weight modification; The node feature masking strategy is as follows: based on the node saliency score, node features that have a greater impact on the saliency score map are retained, while only those node features with a smaller impact are randomly masked. The strategy for changing edge weights is to replace the original weight of the corresponding edge in the adjacency matrix with the significance score of the edge, which is 1 in the original adjacency matrix.

2. The method for training multimodal temporal astronomical data based on graph contrast learning according to claim 1, characterized in that: Multimodal astronomical data includes at least: image data, spectral data, star catalog data and text data. In step one, after collecting the above multimodal astronomical data, data alignment and time division are performed first, and the astronomical image data and spectral data are divided into multiple time windows or time points according to the observation timestamp. Embedding vectors for image patches, continuous bands of spectra, star catalog records, and text fragments or key concepts are obtained by preprocessing the four types of multimodal data, and these vectors are used as the basic units of each modality.

3. The method for training multimodal temporal astronomical data based on graph contrast learning according to claim 2, characterized in that: The construction process of the multimodal astronomical time series data mapping model in step two is as follows: S2.1: The basic units in each modal data are abstracted into nodes in the graph. By calculating the similarity between the features carried by the nodes, the connection relationship between the nodes is constructed, i.e., the edge, thus generating a corresponding graph structure for each modal data. S2.2: For image data and spectral data with time-series attributes, further construct the data according to the time series, and repeat the above construction process for the data at each time point; S2.3: Generate a series of time evolution graphs reflecting the data state at different times, thus obtaining a dynamic, multimodal graph set.

4. The method for training multimodal temporal astronomical data based on graph contrast learning according to claim 2, characterized in that: The process of constructing a new graph from the unified feature representation in step four is as follows: Each feature vector in the unified feature representation is regarded as a node in graph G. By calculating the similarity between the features of these nodes, the edges of the graph are constructed, thus forming a new graph structure that reflects the relationship between features as the input graph in graph contrastive learning.

5. The method for training multimodal temporal astronomical data based on graph contrast learning according to claim 1, characterized in that: The process of selecting positive and negative sample pairs and calculating the contrast loss in step four is as follows: Positive sample pairs: The embedded representations of the same node in different augmented views, as well as the embedded representations of the neighboring nodes of the anchor node within the same view and between different views, are considered positive sample pairs; Negative sample pairs: The embedding representations of non-neighboring nodes of anchor nodes within the same view and the embedding representations of non-neighboring nodes of anchor nodes between different views are considered as negative sample pairs; Contrastive loss is designed by bringing positive sample pairs closer together and pushing negative sample pairs further apart.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-5.

7. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-5.

Images