An astronomical transient source identification method with dynamic update of a multi-modal model

Abstract

The present application relates to the field of astronomical transient source identification combining astronomy and artificial intelligence, and when performing the task of identifying transient sources in the massive data generated by a large-field high-sampling-rate telescope, the existing methods usually use a single classification model relying on image data only, which is difficult to cope with complex and changeable data. The present application proposes an astronomical transient source identification method with dynamic updating of a multi-modal model, which constructs a multi-modal model for classification by combining image data, point spread functions and physical information; combined with artificial review, selects samples with classification errors and possible transient source candidates, and stores them as a new sample set; by injecting the new sample set into the original training set and loading the last trained weights for retraining, the dynamic adjustment and optimization of the model are realized. This method can effectively cope with the massive data generated by a large-field high-sampling-rate telescope, and ensure that the model always maintains efficient and accurate identification performance under complex background and weak signals.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of real-time astronomical transient source identification combined with astronomy and artificial intelligence, and more particularly, to a multi-modal model dynamic updating astronomical transient source identification method. BACKGROUND

[0002] The rapid development of modern astronomical observation equipment, especially the widespread application of large field of view high sampling rate telescopes, enables astronomers to capture extremely short and rapidly changing astronomical phenomena. These devices produce massive data streams every night, and the large amount of data and high update frequency pose great challenges to the identification of astronomical transient sources. Traditional transient source identification methods mainly rely on single image data, and usually use convolutional neural networks to extract and classify image features. This kind of method can only achieve results in small field of view, image feature difference obvious scene, but has obvious limitations in large field of view observation environment. First, in large field of view observation, the field of view of the telescope covers a wide range, and the features of different fields of view of the image are different, especially the point spread function of different fields of view changes greatly, which leads to uneven image features, and further affects the recognition accuracy of the model. Secondly, the information of single image mode is not enough to fully describe the characteristics of transient sources, especially under the influence of observation equipment performance, weather conditions and imaging system, etc., the dark and weak targets in the image are difficult to be effectively captured and identified. In addition, as the observation time of the telescope passes, the change of observation conditions and the change of sky environment may lead to the change of image quality, and the adaptability and accuracy of the observation model are difficult to guarantee, which leads to the decline of the identification performance, and when dealing with observation data with new characteristics or complex background, the observation model may make mistakes. Single model cannot maintain good performance when facing new data. The traditional method makes the model unable to adjust flexibly when facing new data, which leads to the limitation of the model performance when facing new data. SUMMARY

[0003] In view of the defects in the prior art, the purpose of the present application is to provide a multi-modal model dynamic updating astronomical transient source identification method. The present application combines image data, point spread function and physical information, and introduces a visualization and artificial review mechanism to construct a multi-modal classification network with dynamic adjustment and updating capability. This method can maintain high and stable identification performance in the processing of massive data generated by large field of view high sampling rate observation equipment.

[0004] To achieve the above purpose, the present application provides the following technical scheme:

[0005] The application discloses an astronomical transient source identification method based on dynamic updating of a multi-modal model, and the method receives image data, a point spread function and physical information through the multi-modal model, splices features extracted from the image data, features extracted from the point spread function and features extracted from the physical information, uses a classification layer of the multi-modal model to complete classification and displays a classification result through a visual interface, combines manual review to select samples with classification errors and possible transient source candidates as new samples to be stored into an original training sample set, and uses the new samples to update the original training sample set; the multi-modal model is evaluated by using a set threshold value, and when a recall rate or a precision rate of the multi-modal model is less than the threshold value, the new sample set is injected into the original training sample set, a weight of the last time before the new sample set is injected is loaded as a pre-training weight to be retrained, and the performance of the multi-modal model is required to reach the threshold value, so that dynamic updating and optimization of the multi-modal model are realized, and the accuracy and reliability of transient source identification are improved.

[0006] Step 1. Constructing an original training sample set: residual star images obtained by subtracting a template image from an observation image are used as negative samples for training of the multi-modal model; according to imaging characteristics of a telescope system, performance of observation equipment and influence of atmospheric turbulence, simulated images with simulated star images are generated, and simulated star images obtained by subtracting the template image from the simulated images are used as positive samples for training of the multi-modal model;

[0007] Step 2. Constructing a multi-modal model: the multi-modal model includes image feature extraction, point spread function feature extraction and physical information feature extraction, three independent feature vectors obtained through the image feature extraction, the point spread function feature extraction and the physical information feature extraction are spliced in the same dimension to form a multi-modal feature vector, and the multi-modal feature vector is transmitted to a classification layer composed of a full connection layer, a ReLu activation function and a Sigmod activation function to perform classification and obtain a classification probability value;

[0008] Step 3. Visualizing a classification result and manually reviewing: the classification result of the multi-modal model is visually processed and manually reviewed, the classification result is stored into a database, and the database is displayed through a front-end page and interacts with a user;

[0009] Step 4. Dynamically adjusting the multi-modal model: the performance of the multi-modal model is evaluated by calculating a recall rate and a precision rate of the multi-modal model on a verification set.

[0010] Further, the convolutional neural network comprises four convolutional layers, each of which is followed by a ReLU activation function and a dropout layer to enhance the generalization ability of the multi-modal model, and a max-pooling layer is used in the first two convolutional layers to reduce the size of a feature map, and finally the extracted features are flattened into a one-dimensional vector.

[0011] Further, the multi-modal model uses a convolutional neural network with ResNet50 as the backbone for feature extraction, and replaces the last fully connected layer of the convolutional neural network ResNet50 with an identity mapping; the input of the point spread function feature extraction part is the point spread function of the observed image, and the point spread function features are extracted through the convolutional neural network; the physical information is extracted by a feature extraction network composed of a fully connected layer and a ReLu activation function, and the extracted feature vectors are mapped to a higher dimensional space through a step-by-step expansion fully connected layer, and finally a high-dimensional feature vector is generated.

[0012] Further, the physical information is obtained by querying the Gaia galaxy probe satellite catalog, the center position is determined according to the astronomical coordinates RA and Dec of the target to be classified, the radius of the query area is set, and the celestial body information within the set radius range is queried in the Gaia third batch data GaiaDR3.

[0013] Further, artificial review selects two types of samples according to the visualization results: the first type is the sample with classification error, which is marked by the multi-modal model automatically comparing the predicted label and the true label of the sample, and the sample with classification error is marked for artificial review; the second type is a sample that is most likely to be a transient source, which has features including regular circular target outline and continuous appearance in multiple frames of images, combined with apparent magnitude Mag, full width at half maximum FWHM, frame appearance count Count, and confidence level Confidence, to further determine whether it meets the characteristics of a transient source through artificial review; and the two types of samples are corrected and stored in the original training sample set to update the original training sample set.

[0014] Further, the threshold is set to 97-99%.

[0015] Further, in step 4, the multi-modal model dynamically adjusts as follows: when the recall or precision of the multi-modal model does not reach the set threshold or decreases, the new samples stored in step 3 are injected into the original training sample set, the training sample set is expanded and updated, and the weights of the last training of the multi-modal model before updating the sample set are loaded as pre-training weights for retraining; after training, the visualization and artificial review process of step 3 is repeated, and the performance evaluation of the multi-modal model is performed again, and the cycle continues until the performance of the multi-modal model meets the requirements.

[0016] In summary, the application has the following advantages:

[0017] The present application generates a simulated image with a simulated star image by combining the imaging characteristics of a telescope system, the performance of an observation device and the influence of atmospheric turbulence, subtracts the simulated image from a template image using image subtraction technology, and uses the simulated star image obtained after the subtraction as the positive sample of the original training sample set. The simulated star image reflects different morphologies of the target in different fields of view and surrounding environmental information. The different morphologies reflect different contour information of the target at different positions in the image, and the surrounding environmental information reflects the relevance of the target and other stars or galaxies around it. The point spread function and the physical information provide more comprehensive feature information for the multi-modal model.

[0018] The present application adopts a multi-modal network architecture to efficiently combine physical information, a point spread function and image data, so that the multi-modal classification model can more comprehensively understand the characteristics of astronomical transient sources, thereby significantly improving the recognition accuracy.

[0019] The present application introduces classification result visualization and manual review, which uses an intuitive page to make the classification result directly identifiable and verifiable by manual identification, thereby quickly and accurately discovering classification errors and potential transient source candidates, making up for the shortcomings of the multi-modal model in automatic classification, and continuously adjusting the original training sample set through manual review to adjust the multi-modal model.

[0020] The present application continuously introduces new training samples and re-trains the multi-modal model through dynamic updating of the multi-modal model, adjusts the multi-modal model in real time according to actual observations, and ensures the stability and accuracy of the multi-modal model in long-term use. Combined with the manual review results and the dynamic adjustment of the multi-modal model, the training efficiency of the multi-modal model is improved, and the recognition ability of the multi-modal model under complex background and weak signal target is enhanced, ensuring that the multi-modal model can always maintain efficient and accurate transient source recognition ability when facing large-scale and high-complexity astronomical data. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 The present application is a method flowchart.

[0022] Figure 2 The present application is a multi-modal model structure diagram.

[0023] Figure 3 The present application is a curve comparison diagram of the classification effects of the multi-modal model and the single-modal model. DETAILED DESCRIPTION

[0024] The present application will be further described in detail below with reference to the accompanying drawings.

[0025] As Figure 1 and Figure 2As shown, the application discloses a kind of astronomical transient source identification method of multi-modal model dynamic update, including multi-modal model, visualization and artificial review and model dynamic adjustment three parts;Using image subtraction technique, the residual star image obtained by subtracting observed image from template image is used as negative sample for multi-modal model training;According to the imaging characteristics of telescope system, observation equipment performance and atmospheric turbulence influence, generate simulated image with simulated star image, using image subtraction technique, the simulated star image obtained by subtracting simulated image from template image is used as positive sample for multi-modal model training.Negative sample and positive sample constitute original training sample set.Multi-modal model simultaneously receives image data, point spread function PSF and the physical information of surrounding star or galaxy as input, and the features extracted from image data, the features extracted from point spread function and the features extracted from physical information are spliced on the same dimension, point spread function represents contour information, physical information represents surrounding environment information, point spread function and physical information provide more comprehensive feature information for multi-modal model, the spliced features are transmitted to the classification layer of multi-modal model for classification, the feature splicing layer is connected to the classification layer, the classification layer includes fully connected layer and activation function, and then transmitted to the classification layer of multi-modal model for classification;The classification result output by classification layer is presented through visualization interface, artificial review selects classification error sample and possible transient source candidate, and stores classification error sample and possible transient source candidate as new sample to original training sample set, updates original training sample set through new sample, and the updated sample set is original training sample set+new sample set;Transient source candidate is generated in data processing, which may be transient source or not, as candidate, whether it is transient source is confirmed subsequently;Performance evaluation is carried out on multi-modal model, when the recall rate or precision rate of multi-modal model is less than the set threshold, it indicates that multi-modal model is not up to standard, new sample set is injected into original training sample set, the weight of last training before new sample injection is loaded as pre-training weight for retraining, after training, repeat the process of visualization and artificial review and multi-modal model performance evaluation, until the performance of multi-modal model meets the requirements, realize the dynamic update and optimization of multi-modal model, the threshold range is 97-99%, in the embodiment, the threshold is set to 98%, in actual application, the threshold can be adjusted according to different requirements.Transient source: real, final astronomical target to be identified

[0026] Step 1. Constructing the original training sample set: using image subtraction technology, the residual star image obtained by subtracting the template image from the observation image is used as the negative sample for training the multi-modal model; according to the imaging characteristics of the telescope system, the performance of the observation equipment and the influence of atmospheric turbulence, the point spread function of different fields of view of the telescope is extracted, the principal component of the point spread function is extracted by using PCA dimension reduction technology, the principal component is fitted with the star image of the observation image, the simulation star image without background noise is generated, the simulation star image without background noise is added to the observation image to generate a simulated image with simulation star image, using image subtraction technology, the simulation star image obtained by subtracting the simulated image from the template image is used as the positive sample for training the multi-modal model.

[0027] Step 2. Building a multi-modal model: the multi-modal model includes three parts of image feature extraction, point spread function feature extraction and physical information feature extraction. The input of the image feature extraction part is a three-channel image, which corresponds to the observation image, the template image and the image obtained by subtracting the template image from the observation image, respectively. The convolutional neural network with ResNet50 as the backbone is used for feature extraction, and the last fully connected layer of the convolutional neural network ResNet50 is replaced by the identity mapping (Identity) to preserve rich image information. The input of the point spread function feature extraction part is the point spread function of the observation image, which is extracted by the convolutional neural network. The convolutional neural network (CNN) is used as the backbone network to extract the features of the point spread function of the observation image. Through multiple layers of convolution, ReLU activation, pooling and dropout operations, the spatial features of the point spread function are gradually extracted and compressed. The convolutional neural network contains four convolutional layers, each followed by a ReLU activation function and a dropout layer to enhance the generalization ability of the multi-modal model. Max pooling layers are used in the first two convolutional layers to reduce the size of the feature map. Finally, the extracted features are flattened into a one-dimensional vector for subsequent processing.

[0028] The physical information is obtained by querying the Gaia Star Catalog, and the center position of the query is determined according to the astronomical coordinates RA (right ascension) and Dec (declination) of the target to be classified. The radius of the query area can be adjusted according to the observation requirements. In this embodiment, a query area with a radius of 120 arcsec is set, and the celestial body information within this range is queried in the Gaia DR3 or the latest star catalog data. The physical information includes the number of celestial bodies, the star magnitude, the type of celestial body, and the distance from the target. The physical information is extracted by three fully connected layers and three ReLu activation functions. The first fully connected layer maps the input 4-dimensional feature to a 64-dimensional feature space, the second fully connected layer maps the 64-dimensional feature to a 256-dimensional feature space, and the third fully connected layer maps the 256-dimensional feature to a 1024-dimensional feature space. Each fully connected layer is followed by a ReLu activation function. Each fully connected layer maps the low-dimensional feature to a higher-dimensional space, and each ReLu activation function introduces a nonlinear transformation. After the input 4-dimensional feature is processed by the three fully connected layers and the three ReLu activation functions, a 1024 high-dimensional feature vector is finally generated, providing a basis for subsequent multi-modal feature concatenation and classification tasks.

[0029] The independent feature vectors from the three different branches are concatenated in the same dimension to form a multi-modal feature vector. The three fully connected layers are used, the first fully connected layer maps the input 5120-dimensional feature to a 1024-dimensional feature space, the second fully connected layer maps the input 1024-dimensional feature to a 64-dimensional feature space, and each fully connected layer is followed by a ReLu activation function. The first two fully connected layers gradually extract key information from high-dimensional input features, and the ReLu activation function after each fully connected layer introduces a nonlinear transformation, enabling the feature extraction network to learn more complex and abstract feature representations. The third fully connected layer maps the input 64-dimensional feature to a 1-dimensional feature space, and the Sigmoid activation function is used. The third fully connected layer maps the 64-dimensional feature extracted by the first two fully connected layers to a 1-dimensional scalar output, and the Sigmoid activation function further converts the scalar output to a classification probability value between 0 and 1, i.e. the confidence. By setting a confidence threshold, it is determined whether it is a transient source candidate. If the confidence is greater than the threshold, the predicted label is 1, otherwise the predicted label is 0. A label of 1 indicates that it belongs to a transient source candidate, and vice versa.

[0030] Step 3. Classification result visualization and manual review: The classification results of the multi-modal model are visualized and manually reviewed, and the classification results are stored in the database, displayed through the front-end page, and interacted with the user. The display page includes the three-channel images of the target, i.e. the observation image, the template image and the image obtained by subtracting the observation image from the template image, and the information of the target, i.e. the right ascension (RA), the declination (DEC), the image coordinate x, the image coordinate y, the apparent magnitude (Mag), the number of consecutive frame appearances (Count), the full width at half maximum (FWHM), the confidence (Confidence), the predicted label and the true label.

[0031] According to the visualization results, the artificial reviewer selects two types of samples: the first type is the misclassified samples, which are misclassified by the multi-modal model, and the second type is the samples that are most likely to be transient sources. The features of the first type include regular circular target outline and continuous appearance in multiple frame difference images. The features of the second type include apparent magnitude (Mag), full width at half maximum (FWHM), consecutive frame appearance number (Count), and confidence (Confidence). The artificial reviewer further judges whether the samples meet the characteristics of transient sources.

[0032] For the first type of samples, the multi-modal model automatically corrects the label to the correct class; for the second type of samples, the label is uniformly marked as positive samples. The two types of samples will be stored as a new sample set for subsequent injection into the original training sample set to update the original training sample set and optimize the performance of the multi-modal model.

[0033] Manual review is part of the dynamic adjustment of the multi-modal model. The visualization results are directly displayed on the page, and the artificial review is the process of selecting the samples from the displayed results and adding them to the original training sample set for retraining to dynamically adjust the multi-modal model.

[0034] Step 4. Dynamic adjustment of multi-modal model: The performance of the multi-modal model is evaluated by calculating the recall and precision of the multi-modal model on the validation set. When the performance of the multi-modal model does not meet the expected or declines, the new samples stored in step 3 are injected into the original training sample set, the training sample set is expanded and updated, and the last trained weight of the multi-modal model is loaded as the pre-trained weight for retraining. After training, the visualization and manual review process of step 3 is repeated, and the performance of the multi-modal model is evaluated again, and the cycle continues until the performance of the multi-modal model meets the requirements.

[0035] Telescopes capture hundreds to thousands of images each night, often over several years. This results in complex, diverse, and rapidly updated image data. Therefore, multimodal models must be able to dynamically update to adapt to new data. Initially, one or several days of data are typically processed to train a model that meets performance requirements. Subsequently, the recall and precision of the multimodal model are evaluated using a validation set. While the multimodal model's performance remains stable and meets requirements, no updates are required. However, when the interval between newly captured data and the original data exceeds 30 days, or when instrument parameters or the observing environment (such as weather conditions) change significantly, the performance of the multimodal model may degrade. In this case, manual review of the classification results is necessary. Samples with incorrect classifications and those that are highly likely to be transient sources are selected as a new sample set and injected into the original training sample set to update the multimodal model.

[0036] Through this iterative dynamic adjustment process, when the new sample set is combined with the original training sample set, the multimodal model loads the weights from the last training session (before the new sample was injected) as pre-trained weights to accelerate training while preserving the learning of the original features. This retraining of the multimodal model allows it to adapt to the new data features and correct any biases in classification. This dynamic adjustment process not only ensures that the multimodal model maintains efficient and accurate performance when processing transient sources, but also continuously adapts to new data and features, ensuring its ability to handle a variety of complex situations in real-world applications.

[0037] like Figure 3 The figure shows a comparison of the ROC curves (Receiver Operating Characteristic Curve) of the multimodal model and the unimodal model. The ROC curve is a tool for evaluating the performance of a classification model, which shows the relationship between the true positive rate (True Positive Rate) and the false positive rate (False Positive Rate) of the model at different classification thresholds. The closer the curve is to the upper left corner, the better the model performance, and the area under the curve (AUC, Area Under Curve) is used to quantify the classification ability of the model. The closer the AUC value is to 1, the better the classification effect of the model. As can be seen from the figure, the curve of the multimodal model (orange) is closer to the upper left corner than the curve of the unimodal model (blue), and its AUC value (0.993) is significantly higher than the AUC value of the unimodal model (0.978), showing better classification performance. This shows that the multimodal model has higher accuracy and robustness.

[0038] The above merely describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-described embodiments. Any technical solution falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that, for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application shall also be considered as falling within the protection scope of the present application.

Claims

1. A method for identifying astronomical transient sources with dynamic updating of a multimodal model, characterized in that: Image data, point spread function and physical information are received through a multimodal model. After the features extracted from the image data, the features extracted from the point spread function and the features extracted from the physical information are spliced, the classification layer of the multimodal model is used to complete the classification and the classification results are displayed through a visual interface. Combined with manual review, samples with classification errors and possible transient source candidates are selected as new samples and stored in the original training sample set. The original training sample set is updated using the new samples. The multimodal model is evaluated using a set threshold. If the recall or precision of the multimodal model is less than the threshold, the new sample set is injected into the original training sample set, and the weights of the last training before the new sample injection are loaded as pre-training weights for retraining until the performance of the multimodal model meets the threshold requirement, thereby realizing dynamic updating and optimization of the multimodal model and improving the accuracy and reliability of transient source identification. The specific steps include: Step 1. Construct the original training sample set: Using image subtraction technology, the residual star image obtained by subtracting the observed image from the template image is used as the negative sample for multimodal model training; based on the imaging characteristics of the telescope system, the performance of the observation equipment, and the influence of atmospheric turbulence, a simulated image with simulated star images is generated. Using image subtraction technology, the simulated star image obtained by subtracting the simulated image from the template image is used as the positive sample for multimodal model training; Step 2. Construct a multimodal model: The multimodal model includes image feature extraction, point spread function feature extraction, and physical information feature extraction. The three independent feature vectors extracted through image feature extraction, point spread function feature extraction, and physical information feature extraction are concatenated on the same dimension to form a multimodal feature vector. The vector is then passed to the classification layer composed of a fully connected layer, ReLu activation function, and Sigmund activation function for classification, and the classification probability value is obtained. Step 3. Visualization and manual review of classification results: Visualize and manually review the classification results of the multimodal model, store the classification results in the database, and display them on the front-end page for user interaction; Step 4. Dynamic adjustment of the multimodal model: Evaluate the performance of the multimodal model by calculating the recall and precision of the multimodal model on the validation set.

2. The astronomical transient source identification method with dynamic updating of multimodal models according to claim 1, characterized in that: The multimodal model uses a convolutional neural network with ResNet50 as the backbone for feature extraction, and replaces the last fully connected layer of the convolutional neural network ResNet50 with an identity mapping; the input of the point spread function feature extraction part is the point spread function of the observed image, and the point spread function features are extracted through the convolutional neural network; physical information is extracted through a feature extraction network composed of a fully connected layer and a ReLu activation function, and the extracted feature vector is mapped to a higher dimensional space through a gradually expanded fully connected layer, and finally a high-dimensional feature vector is generated.

3. The astronomical transient source identification method with dynamic updating of multimodal models according to claim 2, characterized in that: The convolutional neural network contains four convolutional layers, each followed by a ReLU activation function and a dropout layer to enhance the generalization ability of the multimodal model. A maximum pooling layer is used in the first two convolutional layers to reduce the size of the feature map, and finally the extracted features are flattened into a one-dimensional vector.

4. The astronomical transient source identification method with dynamic updating of multimodal models according to claim 3 is characterized in that: The physical information is obtained by querying the Gaia galaxy exploration satellite star catalog. The query center position is determined according to the astronomical coordinates of the target to be classified, the right ascension RA and the declination Dec, and the radius of the query area is set. The celestial body information within the set radius is queried in the third batch of Gaia data, Gaia DR3.

5. The astronomical transient source identification method with dynamic updating of multimodal models according to claim 1, characterized in that: The manual review selects two types of samples based on the visualization results: the first type is misclassified samples. The multimodal model automatically compares the predicted labels and true labels of the samples, marking the misclassified samples for manual review. The second type is samples that are highly likely to be transient sources. Characteristics include the target outline being a regular circle and appearing continuously in multiple frames. At the same time, combined with the apparent magnitude Mag, full width at half maximum FWHM, number of consecutive frame appearances Count, and confidence, manual review is further used to determine whether they meet the characteristics of transient sources. The two types of samples are corrected and stored in the original training sample set to update the original training sample set.

6. The astronomical transient source identification method with dynamic updating of multimodal models according to claim 1, characterized in that: The threshold is set at 97-99%.

7. The astronomical transient source identification method with dynamic updating of multimodal models according to claim 1, characterized in that: In step 4, the multimodal model dynamic adjustment method is as follows: when the recall or precision of the multimodal model does not reach the set threshold or decreases, the new samples stored in step 3 are injected into the original training sample set, the training sample set is expanded and updated, and the weights of the multimodal model last trained before the sample set is updated are loaded as pre-training weights for retraining; After training is completed, repeat the visualization and manual review process in step 3 and perform multimodal model performance evaluation again. Continue the cycle until the multimodal model performance meets the requirements.

Images