Astronomical transient source identification method for dynamically updating multi-modal model

Abstract

The invention relates to the field of astronomy transient source recognition combining astronomy and artificial intelligence, and when a transient source recognition task in mass data shot by a large-view-field high-sampling-rate telescope is carried out, an existing method generally uses a single classification model only depending on image data and is difficult to deal with complex and changeable data. The invention provides an astronomical transient source identification method for dynamically updating a multi-modal model. The method comprises the following steps: constructing the multi-modal model for classification by combining image data, a point spread function and physical information; and in combination with manual examination, selecting samples with wrong classification and possible transient source candidates, storing the samples and the possible transient source candidates as a new sample set, injecting the new sample set into the original training set, and loading the weight of the last training for retraining, so that dynamic adjustment and optimization of the model are realized. The method can effectively cope with mass data generated by a telescope with a large field of view and a high sampling rate, and ensures that the model always keeps efficient and accurate recognition performance under a complex background and weak signals.

Description

Technical Field

[0001] The present invention relates to the field of real-time astronomical transient source identification combining astronomy and artificial intelligence, and more specifically, to an astronomical transient source identification method with dynamic updating of a multi-modal model. Background Art

[0002] The rapid development of modern astronomical observation equipment, especially the widespread use of large-field-of-view and high-sampling-rate telescopes, has enabled astronomers to capture extremely short-lived and rapidly changing astronomical phenomena. These devices generate massive data streams every night. The large amount of data and high update frequency pose great challenges to the identification of astronomical transient sources. Traditional transient source identification methods mainly rely on single image data, and usually use convolutional neural networks to extract and classify image features. This type of method can only achieve results in scenes with small fields of view and obvious differences in image features, but has significant limitations in large-field-of-view observation environments. First, in large-field-of-view observations, the field of view of the telescope covers a wide range, and the features of different fields of view of the image are different, especially the point spread functions of different fields of view vary greatly, resulting in uneven image features, which in turn affects the recognition accuracy of the model. Secondly, the information of a single image modality is not enough to fully describe the characteristics of transient sources. Especially under the influence of factors such as observation equipment performance, weather conditions and imaging systems, faint targets in the image are difficult to be effectively captured and identified. In addition, as the telescope takes images over time, changes in observation conditions and sky environment may lead to changes in image quality. The adaptability and accuracy of the observation model are difficult to guarantee, resulting in reduced recognition performance. When processing observation data with new features or complex backgrounds, the observation model may misjudge. A single model cannot maintain good performance on new data. Traditional methods make it impossible for the model to adjust flexibly when faced with new data, resulting in limited model performance when faced with new data. Summary of the invention

[0003] In view of the shortcomings of the prior art, the purpose of the present invention is to provide an astronomical transient source identification method with a dynamically updated multimodal model. The invention constructs a multimodal classification network with dynamic adjustment and update capabilities by combining image data, point spread function and physical information, and introducing visualization and manual review mechanisms. This method can always maintain high accuracy and stable recognition performance in the processing of massive data generated by large-field-of-view and high-sampling-rate observation equipment.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A method for identifying astronomical transient sources with dynamic updating of a multimodal model, receiving image data, point spread function and physical information through a multimodal model, concatenating features extracted from image data, features extracted from point spread function and features extracted from physical information, using a classification layer of the multimodal model to complete classification and display the classification results through a visual interface, selecting misclassified samples and possible transient source candidates as new samples in combination with manual review and storing them in an original training sample set, and using the new samples to update the original training sample set; evaluating the multimodal model using a set threshold, if the recall or precision of the multimodal model is less than the threshold, injecting the new sample set into the original training sample set, loading the weights of the last training before the injection of the new sample as pre-training weights for retraining until the performance of the multimodal model reaches the threshold requirement, realizing dynamic updating and optimization of the multimodal model, and improving the accuracy and reliability of transient source identification, specifically comprising the following steps:

[0006] Step 1. Construct the original training sample set: Use the image subtraction technology to subtract the residual star image obtained by subtracting the observed image from the template image as the negative sample for multimodal model training; Generate a simulated image with simulated star images based on the imaging characteristics of the telescope system, the performance of the observation equipment and the influence of atmospheric turbulence, and use the image subtraction technology to subtract the simulated image from the template image to obtain the simulated star image as the positive sample for multimodal model training;

[0007] Step 2. Construct a multimodal model: The multimodal model includes image feature extraction, point spread function feature extraction and physical information feature extraction. The three independent feature vectors extracted by image feature extraction, point spread function feature extraction and physical information feature extraction are concatenated on the same dimension to form a multimodal feature vector, which is then passed to the classification layer composed of the full connection layer, ReLu activation function and Sigmod activation function for classification to obtain the probability value of the classification.

[0008] Step 3. Visualization and manual review of classification results: Visualize and manually review the classification results of the multimodal model, store the classification results in the database, display them through the front-end page and interact with users;

[0009] Step 4. Dynamic adjustment of the multimodal model: The performance of the multimodal model is evaluated by calculating the recall and precision of the multimodal model on the validation set.

[0010] Furthermore, the convolutional neural network contains four convolutional layers, each of which is followed by a ReLU activation function and a dropout layer to enhance the generalization ability of the multimodal model, and a maximum pooling layer is used in the first two convolutional layers to reduce the size of the feature map. Finally, the extracted features are flattened into a one-dimensional vector.

[0011] Furthermore, the multimodal model uses a convolutional neural network with ResNet50 as the backbone for feature extraction, and replaces the last fully connected layer of the convolutional neural network ResNet50 with an identity mapping; the input of the point spread function feature extraction part is the point spread function of the observed image, and the point spread function features are extracted through the convolutional neural network; the physical information is extracted through a feature extraction network composed of a fully connected layer and a ReLu activation function, and the extracted feature vector is mapped to a higher dimensional space through a gradually expanded fully connected layer, and finally a high-dimensional feature vector is generated.

[0012] Furthermore, physical information is obtained by querying the Gaia galaxy exploration satellite star catalog, determining the query center position according to the astronomical coordinates of the target to be classified, the radius of the query area is set, and the celestial body information within the set radius is queried in the third batch of Gaia data GaiaDR3.

[0013] Furthermore, two types of samples are selected by manual review based on the visualization results: the first type is misclassified samples. The predicted labels and true labels of the samples are automatically compared through the multimodal model, and the misclassified samples are marked for manual review; the second type is samples that are very likely to be transient sources. Characteristics include the target outline being a regular circle and appearing continuously in multiple frames of images. At the same time, combined with the apparent magnitude Mag, full width at half maximum FWHM, number of occurrences in subsequent frames Count, and confidence Confidence, manual review is used to further determine whether they meet the characteristics of transient sources; and the two types of samples are corrected and stored in the original training sample set to update the original training sample set.

[0014] Furthermore, the threshold is set to 97-99%.

[0015] Furthermore, in step 4, the dynamic adjustment method of the multimodal model is as follows: when the recall rate or precision rate of the multimodal model does not reach the set threshold or decreases, the new samples stored in step 3 are injected into the original training sample set, the training sample set is expanded and updated, and the weights of the multimodal model last trained before the sample set is updated are loaded as pre-training weights for retraining; after the training is completed, the visualization and manual review process of step 3 is repeated, and the multimodal model performance is evaluated again, and the cycle continues until the multimodal model performance meets the requirements.

[0016] In summary, the invention has the following beneficial effects:

[0017] The present invention combines the imaging characteristics of the telescope system, the performance of the observation equipment and the influence of atmospheric turbulence to generate a simulated image with simulated star images. The simulated star images obtained by subtracting the simulated image from the template image are used as positive samples of the original training sample set by using image subtraction technology. The simulated star images reflect the different forms of the target in different fields of view and the surrounding environmental information. Among them, the different forms reflect the different contour information of the target at different positions in the image, and the surrounding environmental information reflects the correlation between the target and other stars or galaxies around it. The point spread function and physical information provide more comprehensive feature information for the multimodal model.

[0018] The present invention adopts a multimodal network architecture to achieve an efficient combination of physical information, point spread function and image data, so that the multimodal classification model can more comprehensively understand the characteristics of astronomical transient sources, thereby significantly improving the recognition accuracy.

[0019] The present invention introduces classification result visualization and manual review, and utilizes an intuitive page to allow classification results to be directly identified and verified manually, thereby quickly and accurately discovering classification errors and potential transient source candidates, making up for the shortcomings of the multimodal model in automatic classification. The original training sample set is continuously adjusted through manual review to achieve adjustment of the multimodal model.

[0020] The present invention continuously introduces new training samples and retrains the multimodal model through dynamic updating of the multimodal model, and adjusts the multimodal model in real time according to actual observations, thereby ensuring the stability and accuracy of the multimodal model in long-term use. Combining the manual review results with the dynamic adjustment of the multimodal model not only improves the training efficiency of the multimodal model, but also enhances the recognition ability of the multimodal model under complex backgrounds and weak signal targets, ensuring that when facing large-scale, highly complex astronomical data, it can always maintain efficient and accurate transient source recognition capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The figure is a flow chart of the method of the present invention.

[0022] Figure 2 This is a structural diagram of the multimodal model in the present invention.

[0023] Figure 3 It is a curve comparison diagram of the classification effects of the multimodal model and the unimodal model in the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0025] like Figure 1 and Figure 2As shown, the present invention discloses an astronomical transient source identification method with dynamic updating of multimodal model, including three parts: multimodal model, visualization and manual review, and dynamic adjustment of model; using image subtraction technology, the residual star image obtained by subtracting the observed image from the template image is used as a negative sample for multimodal model training; according to the imaging characteristics of the telescope system, the performance of the observation equipment, and the influence of atmospheric turbulence, a simulated image with simulated star images is generated, and using image subtraction technology, the simulated star image obtained by subtracting the simulated image from the template image is used as a positive sample for multimodal model training. Negative samples and positive samples constitute the original training sample set. The multimodal model simultaneously receives image data, point spread function PSF and physical information of surrounding stars or galaxies as input, and splices the features extracted from the image data, the features extracted from the point spread function and the features extracted from the physical information in the same dimension. The point spread function represents contour information, and the physical information represents the surrounding environmental information. The point spread function and the physical information provide more comprehensive feature information for the multimodal model. The spliced ​​features are passed to the classification layer of the multimodal model for classification. The feature splicing layer is connected to the classification layer, which includes a fully connected layer and an activation function, and then passed to the classification layer of the multimodal model to complete the classification. The classification results output by the classification layer are presented through a visual interface, and the misclassified samples and possible transient source candidates are manually reviewed and selected, and the misclassified samples and possible transient source candidates are stored as new samples in the original training sample set. , the original training sample set is updated through new samples, and the updated sample set is the original training sample set + new sample set; the transient source candidate is generated in the data processing, which may be a transient source or not. It is used as a candidate to be confirmed whether it is a transient source later; the multimodal model is evaluated for performance. When the recall rate or precision rate of the multimodal model is less than the set threshold, it means that the multimodal model does not meet the standard. The new sample set is injected into the original training sample set, and the weight of the last training before the new sample injection is loaded as the pre-training weight for retraining. After the training is completed, the visualization and manual review and multimodal model performance evaluation process are repeated until the multimodal model performance meets the requirements, and the dynamic update and optimization of the multimodal model are realized. The threshold range is 97-99%. In this embodiment, the threshold is set to 98%. In actual application, the threshold can be adjusted according to different needs. Transient source: the real astronomical target that is ultimately intended to be identified. The present invention improves the accuracy and reliability of the identification of transient sources, i.e., real astronomical targets, and specifically includes the following steps:

[0026] Step 1. Construct the original training sample set: use the image subtraction technology to subtract the observed image from the template image to obtain the residual star image as the negative sample for multimodal model training; extract the point spread function of the telescope in different fields of view according to the imaging characteristics of the telescope system, the performance of the observation equipment and the influence of atmospheric turbulence, use the PCA dimensionality reduction technology to extract the principal components of the point spread function, fit the principal components with the star image of the observed image to generate a simulated star image without background noise, add the simulated star image without background noise to the observed image to generate a simulated image with simulated star images, and use the image subtraction technology to subtract the simulated image from the template image to obtain the simulated star image as the positive sample for multimodal model training.

[0027] Step 2. Build a multimodal model: The multimodal model includes three parts: image feature extraction, point spread function feature extraction, and physical information feature extraction. The input of the image feature extraction part is a three-channel image, which corresponds to the observed image, the template image, and the image obtained by subtracting the observed image from the template image. The convolutional neural network with ResNet50 as the backbone is used for feature extraction, and the last fully connected layer of the convolutional neural network ResNet50 is replaced with the identity mapping (Identity) to retain rich image information. The input of the point spread function feature extraction part is the point spread function of the observed image. The point spread function features are extracted through the convolutional neural network. The convolutional neural network (CNN) is used as the backbone network to extract the features of the point spread function of the observed image. Through multi-layer convolution, ReLU activation, pooling, and Dropout operations, the spatial features of the point spread function are gradually extracted and compressed. The convolutional neural network contains four convolutional layers, each followed by a ReLU activation function and a dropout layer to enhance the generalization ability of the multimodal model. The maximum pooling layer is used in the first two convolutional layers to reduce the size of the feature map. Finally, the extracted features are flattened into a one-dimensional vector for subsequent processing.

[0028] The physical information is obtained by querying the Gaia Star Catalog, and the query center position is determined according to the astronomical coordinates RA (right ascension) and Dec (declination) of the target to be classified. The radius of the query area can be adjusted according to the observation requirements. In this embodiment, a query area with a radius of 120 arcsec is set, and the celestial body information within this range is queried in the third batch of Gaia data (GaiaDR3) or the latest star catalog data. The queried physical information includes the number of celestial bodies, the magnitude, the category of celestial bodies, and the distance from the celestial body to the target. The physical information is extracted through three fully connected layers and three ReLu activation functions. The first fully connected layer maps the input 4-dimensional features to a 64-dimensional feature space, the second fully connected layer maps the 64-dimensional features to a 256-dimensional feature space, and the third fully connected layer maps the 256-dimensional features to a 1024-dimensional feature space. Each fully connected layer passes through the ReLu activation function. Each fully connected layer maps low-dimensional features to a higher-dimensional space, and each ReLu activation function introduces a nonlinear transformation. After the input 4-dimensional features pass through three fully connected layers and three ReLu activation functions, a 1024-dimensional feature vector is finally generated, which provides a basis for subsequent multimodal feature concatenation and classification tasks.

[0029] The independent feature vectors from three different branches are concatenated on the same dimension to form a multimodal feature vector. Through three fully connected layers, the first fully connected layer maps the input 5120-dimensional features to a 1024-dimensional feature space, and the first fully connected layer maps the input 1024-dimensional features to a 64-dimensional feature space. Each fully connected layer is activated by the ReLu function. The first two fully connected layers gradually extract key information from the high-dimensional input features, and the ReLU activation function after each fully connected layer introduces nonlinear transformations, enabling the feature extraction network to learn more complex and abstract feature representations. The third fully connected layer maps the input 64-dimensional features to a 1-dimensional feature space. After the Sigmoid activation function, the third fully connected layer maps the 64-dimensional features extracted by the first two fully connected layers to a 1-dimensional scalar output. The Sigmoid activation function further converts the scalar output into a classification probability value between 0 and 1, i.e., confidence. By setting the confidence threshold, we determine whether it is a transient source candidate. If the confidence is greater than the threshold, the output prediction label is 1, otherwise the output prediction label is 0. A label of 1 indicates that it is a transient source candidate, otherwise it does not belong to a transient source candidate.

[0030] Step 3. Visualization and manual review of classification results: Visualize and manually review the classification results of the multimodal model, store the classification results in the database, display them through the front-end page, and interact with the user in a friendly manner. The display page includes three-channel images of the target, namely the observed image, the template image, and the image obtained by subtracting the observed image from the template image, as well as the target information, namely right ascension (RA), declination (DEC), image coordinate x, image coordinate y, apparent magnitude (Mag), number of consecutive frame occurrences (Count), full width at half maximum (FWHM), confidence (Confidence), predicted label, and true label.

[0031] Based on the visualization results, the manual reviewer selected two types of samples: the first type is misclassified samples, which are incorrectly classified in the multimodal model prediction. It may be that negative samples are misclassified as positive samples, or positive samples are classified as negative samples. The multimodal model automatically compares the predicted labels and true labels of the samples, and marks the misclassified samples for manual review; the second type is samples that are very likely to be transient sources. Features include the target outline presenting a regular circle and appearing continuously in multiple frame difference images. At the same time, combined with the apparent magnitude (Mag), full width at half maximum (FWHM), number of consecutive frame occurrences (Count), and confidence (Confidence), manual review is used to further determine whether they meet the characteristics of transient sources.

[0032] For the first type of samples, the multimodal model automatically corrects their labels to the correct category; for the second type of samples, they are uniformly marked as positive samples. The two types of samples will be stored as new sample sets for subsequent injection into the original training sample set to complete the update of the original training sample set and optimize the performance of the multimodal model.

[0033] Manual review is part of the dynamic adjustment of the multimodal model. The visualization result is to intuitively display the classification results of the multimodal model on the page. Manual review is to manually screen the displayed results and add the selected samples to the original training sample set for retraining, so as to dynamically adjust the multimodal model.

[0034] Step 4. Dynamic adjustment of multimodal model: Evaluate the performance of the multimodal model by calculating the recall and precision of the multimodal model on the validation set. When the performance of the multimodal model does not meet expectations or declines, inject the new samples stored in step 3 into the original training sample set, expand and update the training sample set, and load the weights of the last training of the multimodal model as pre-training weights for retraining. After the training is completed, repeat the visualization and manual review process of step 3, and evaluate the performance of the multimodal model again, and continue the cycle until the performance of the multimodal model meets the requirements.

[0035] The number of images taken by telescopes every night ranges from hundreds to thousands, and the shooting time lasts for several years, resulting in complex and diverse image data and rapid updates. Therefore, the multimodal model must have the ability to dynamically update in order to adapt to new data in a timely manner. In the initial stage, one or several days of data are usually processed to train a model with satisfactory performance. Subsequently, the recall and precision of the multimodal model are evaluated through the validation set. When the performance of the multimodal model remains stable and meets the requirements, it does not need to be updated. However, when the time interval between the newly captured data and the original data is more than 30 days or the instrument parameters and observation environment (such as weather conditions) change significantly, the performance of the multimodal model may decline. At this time, it is necessary to review the classification results through manual review, select the misclassified samples and the samples that are most likely to be transient sources as the new sample set, and inject them into the original training sample set for multimodal model update.

[0036] Through the above iterative dynamic adjustment process, after the new sample set is combined with the original training sample set, the multimodal model will load the weights of the last training, that is, the last training before the new sample is injected, as pre-training weights to accelerate training and retain the learning of the original features. Retraining the multimodal model enables the multimodal model to adapt to new data features and correct deviations in classification. After the dynamic adjustment process, the multimodal model can not only maintain efficient and accurate performance when processing transient sources, but also continuously adapt to new data and new features, ensuring that it can cope with various complex situations in practical applications.

[0037] like Figure 3 The figure shows a comparison of the ROC curves (Receiver Operating Characteristic Curve) of the multimodal model and the unimodal model. The ROC curve is a tool for evaluating the performance of a classification model, which shows the relationship between the true positive rate (True Positive Rate) and the false positive rate (False Positive Rate) of the model at different classification thresholds. The closer the curve is to the upper left corner, the better the model performance is, and the area under the curve (AUC, Area Under Curve) is used to quantify the classification ability of the model. The closer the AUC value is to 1, the better the classification effect of the model. As can be seen from the figure, the curve of the multimodal model (orange) is closer to the upper left corner than the curve of the unimodal model (blue), and its AUC value (0.993) is significantly higher than the AUC value of the unimodal model (0.978), showing better classification performance. This shows that the multimodal model has higher accuracy and robustness.

[0038] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for identifying astronomical transient sources with dynamic updating of multimodal models, characterized in that: The image data, point spread function and physical information are received through the multimodal model, and the features extracted from the image data, the features extracted from the point spread function and the features extracted from the physical information are spliced. The classification is completed by using the classification layer of the multimodal model and the classification results are displayed through a visual interface. Combined with manual review, the samples with classification errors and possible transient source candidates are selected as new samples and stored in the original training sample set, and the original training sample set is updated by using the new samples; the multimodal model is evaluated by using the set threshold. If the recall rate or the precision rate of the multimodal model is less than the threshold, the new sample set is injected into the original training sample set, and the weights of the last training before the injection of the new sample are loaded as pre-training weights for retraining until the performance of the multimodal model reaches the threshold requirement, so as to realize the dynamic update and optimization of the multimodal model and improve the accuracy and reliability of transient source identification. Specifically, the following steps are included: Step 1. Construct the original training sample set: Use the image subtraction technology to subtract the residual star image obtained by subtracting the observed image from the template image as the negative sample for multimodal model training; Generate a simulated image with simulated star images based on the imaging characteristics of the telescope system, the performance of the observation equipment and the influence of atmospheric turbulence, and use the image subtraction technology to subtract the simulated image from the template image to obtain the simulated star image as the positive sample for multimodal model training; Step 2. Construct a multimodal model: The multimodal model includes image feature extraction, point spread function feature extraction and physical information feature extraction. The three independent feature vectors extracted by image feature extraction, point spread function feature extraction and physical information feature extraction are concatenated on the same dimension to form a multimodal feature vector, which is then passed to the classification layer composed of the full connection layer, ReLu activation function and Sigmod activation function for classification to obtain the probability value of the classification. Step 3. Visualization and manual review of classification results: Visualize and manually review the classification results of the multimodal model, store the classification results in the database, display them through the front-end page and interact with users; Step 4. Dynamic adjustment of the multimodal model: The performance of the multimodal model is evaluated by calculating the recall and precision of the multimodal model on the validation set.

2. The astronomical transient source identification method with dynamic updating of multi-modal models according to claim 1 is characterized in that: The convolutional neural network includes four convolutional layers, each of which is followed by a ReLU activation function and a dropout layer to enhance the generalization ability of the multimodal model. A maximum pooling layer is used in the first two convolutional layers to reduce the size of the feature map, and finally the extracted features are flattened into a one-dimensional vector.

3. The astronomical transient source identification method with dynamic updating of multi-modal models according to claim 1 is characterized in that: The multimodal model uses a convolutional neural network with ResNet50 as the backbone for feature extraction, and replaces the last fully connected layer of the convolutional neural network ResNet50 with an identity mapping; the input of the point spread function feature extraction part is the point spread function of the observed image, and the point spread function features are extracted through the convolutional neural network; physical information is extracted through a feature extraction network composed of a fully connected layer and a ReLu activation function, and the extracted feature vector is mapped to a higher dimensional space through a gradually expanded fully connected layer, and finally a high-dimensional feature vector is generated.

4. The astronomical transient source identification method with dynamic updating of multi-modal models according to claim 3 is characterized in that: The physical information is obtained by querying the Gaia galaxy detection satellite star catalog, determining the query center position according to the astronomical coordinates of the target to be classified, the radius of the query area is set, and the celestial body information within the set radius is queried in the third batch of Gaia data Gaia DR3.

5. The astronomical transient source identification method with dynamic updating of multi-modal models according to claim 1, characterized in that: The manual review selects two types of samples according to the visualization results: the first type is samples that are misclassified, and the predicted labels and true labels of the samples are automatically compared by the multimodal model, and the misclassified samples are marked for manual review; the second type is samples that are very likely to be transient sources, and the characteristics include the target contour presenting a regular circle and appearing continuously in multiple frames of images. At the same time, combined with the apparent magnitude Mag, the full width at half maximum FWHM, the number of consecutive frame appearances Count, and the confidence Confidence, manual review is performed to further determine whether they meet the transient source characteristics; The two types of samples are corrected and stored in the original training sample set to update the original training sample set.

6. The astronomical transient source identification method with dynamic updating of multi-modal models according to claim 1, characterized in that: The threshold is set to 97-99%.

7. The astronomical transient source identification method with dynamic updating of multi-modal models according to claim 1, characterized in that: In step 4, the multimodal model dynamic adjustment method is as follows: when the recall rate or precision rate of the multimodal model does not reach the set threshold or decreases, the new samples stored in step 3 are injected into the original training sample set, the training sample set is expanded and updated, and the weights of the multimodal model trained last time before the sample set is updated are loaded as pre-training weights for retraining; After training is completed, repeat the visualization and manual review process in step 3 and perform multimodal model performance evaluation again. Continue the cycle until the multimodal model performance meets the requirements.

Images