Image analysis method based on multi-modal large model and AI assistant system

Abstract

The invention relates to an image analysis method based on a multi-modal large model and an AI assistant system, and belongs to the technical field of new-generation information, and the analysis method comprises the steps: inputting image analysis original data into a computer system; the method comprises the following steps: preprocessing original data, dividing a training set and a test set according to a category balance principle, extracting feature images and unifying formats to obtain an annotation data set of a classification task; the annotation data set of the classification task is converted into a visual question and answer VQA prompt suitable for fine tuning of a multi-mode large language model, and a VQA data set is constructed; performing knowledge injection and reasoning model optimization on the multi-modal large language model based on the VQA data set; and calling the multi-modal large-scale language model after knowledge injection and reasoning optimization to complete image analysis, and outputting an analysis result generated by reasoning of the multi-modal large-scale language model. According to the method, the blank of a special data set in the field of radio astronomy can be filled, and the analysis ability and cross-task generalization ability of the model to radio astronomy images are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence and industrial internet, and particularly relates to technical contents such as semantic understanding, knowledge graph, machine question answering and reasoning model, and in particular to an image analysis method based on multi-modal large language models (MLLMs) and an AI assistant system. BACKGROUND

[0002] In recent years, as a key discipline for exploring the mysteries of the universe, radio astronomy relies on large-scale observation facilities such as the Square Kilometre Array (SKA) and the Five-hundred-meter Aperture Spherical Radio Telescope (FAST) to obtain massive data. In recent years, these facilities have driven the explosive growth of radio astronomy data. In the face of such a huge data stream, traditional manual analysis methods are inefficient and cannot meet the scientific research needs of real-time interpretation of celestial signals (such as pulsars, radio bursts, and galaxy morphologies), so there is an urgent need for automated and intelligent image analysis technology support.

[0003] In the field of radio astronomy data processing, machine learning and deep learning technologies have been widely applied. Traditional machine learning models (such as support vector machines and random forests) complete classification tasks by extracting manually designed statistical features (such as celestial flux, period, and dispersion), but they are highly dependent on the experience of domain experts and have poor universality; deep learning models (such as convolutional neural networks and Transformers) can automatically learn image features and have made progress in tasks such as pulsar identification and radio galaxy morphology classification, but they have limitations in task binding. For example, a model designed specifically for pulsar classification cannot be directly adapted to radio burst detection, and it ignores the supplementary value of textual context (such as observation reports) and numerical attributes to the analysis results, and has weak cross-task generalization ability.

[0004] With the development of artificial intelligence technology, multi-modal large language models (MLLMs) have shown great potential in the fields of medicine and optical astronomy due to their ability to integrate multiple sources of information such as text and images. They can not only output classification results but also generate text explanations that conform to human understanding logic. However, the application of such models in radio astronomy is still in its infancy, because general MLLMs have not been trained with knowledge from the field of radio astronomy, and lack understanding of professional features such as pulsar dispersion measurement and radio galaxy lobe structure, resulting in insufficient analysis accuracy of radio astronomy images, and even misjudgment of noise signals. At the same time, existing astronomical data sets are mostly single-task labeled (such as containing only classification labels), and lack visual question answering (VQA) samples that consider both images and text, which cannot support the training and performance evaluation of MLLMs, restricting their application in radio astronomy research.

[0005] Therefore, the current technical system mainly faces three core problems. First, the general MLLMs have poor adaptability to radio astronomy tasks, and it is difficult to simultaneously complete the two core tasks of image classification and visual question answering due to the lack of domain-specific knowledge. Second, there is no standardized VQA dataset in the field of radio astronomy, and the existing data is scattered in different observation projects, with inconsistent formats and lack of multi-modal labeling, which cannot effectively support model training. Third, the existing analysis tools are complex to operate, and there is a shortage of compound talents who understand both astronomy and artificial intelligence, which requires researchers to spend a lot of time learning to use the tools, indirectly reducing the data interpretation efficiency and hindering the knowledge discovery process in the field of radio astronomy. SUMMARY

[0006] The main purpose of the present application is to provide an image analysis method based on a multi-modal large model and an AI assistant system, to fill the gap of specialized datasets in the field of radio astronomy, and to improve the analysis ability and cross-task generalization ability of the model for radio astronomy images.

[0007] Based on the first main aspect of the present application, an image analysis method based on a multi-modal large model is provided, comprising the following steps:

[0008] Inputting image analysis raw data into a computer system deployed with a multi-modal large language model;

[0009] The computer system pre-processes the raw data, divides the training set and the test set according to the category balance principle, extracts the feature images and unifies the format, and obtains the labeled dataset for the classification task;

[0010] The labeled dataset for the classification task is converted into a visual question answering (VQA) prompt suitable for fine-tuning the multi-modal large language model, and a VQA dataset is constructed;

[0011] Based on the VQA dataset, the multi-modal large language model is subjected to knowledge injection and reasoning model optimization, including: using LoRA quantization technology to build a fine-tuning framework, encoding the image into a visual embedding and mapping it to the latent embedding space of the multi-modal large language model, designing a multi-modal prompt template integrating visual markers, text queries and task identifiers, adapting the training parameters according to the model parameter size, making the multi-modal large language model learn and structure the domain knowledge, and optimizing its reasoning mechanism based on multi-modal input;

[0012] The multi-modal large language model subjected to knowledge injection and reasoning optimization is called, the to-be-analyzed image and the corresponding prompt template are fused into a multi-modal input, the learned domain knowledge and the optimized reasoning model of the multi-modal large language model are called through the multi-modal large language model, multi-step logical deduction is performed, image analysis is completed, and the analysis results generated by the reasoning of the multi-modal large language model are outputted.

[0013] As a further preferred scheme, in the foregoing method, the image analysis raw data includes four types of open-source radio astronomy raw images and associated annotation information, and the annotation information includes astrophysical parameters and observation feature descriptions.

[0014] The four types of open-source radio astronomy raw data include: pulsar candidate images and labeled tags of the FAST dataset, pulsar candidate images and physical property parameters of the HTRU Medlat dataset, solar radio spectrum sample images and subtitle information of the Spectrumcls dataset, and radio galaxy source images and morphological annotations of the Radio Galaxy dataset.

[0015] As a further preferred scheme, in the foregoing method, when the raw data is preprocessed, the feature images are extracted, including: extracting profile graphs, phase graphs, sub-band graphs, and dispersion measurement graphs from FAST dataset samples; extracting three phase graphs, sub-band graphs, and velocity-momentum graphs from HTRU Medlat dataset samples; and uniformly adjusting the sample images of the Spectrumcls dataset to a preset resolution.

[0016] As a further preferred scheme, in the foregoing method, the labeled dataset of the classification task is converted into a VQA prompt suitable for fine-tuning a multi-modal large language model, including:

[0017] The image input is designed in three modes: single, multiple, and combination; in the single mode, only a single frame image is used as input; in the multiple mode, all available images are input into the visual encoder one by one, and the features are spliced to complete the projection; in the combination mode, all available images are integrated into a single input image and sent to the visual encoder;

[0018] Numerical features and empirical rules are embedded in the prompt template to determine the optimal feature set for multi-modal large language model input and integrate domain knowledge into task processing;

[0019] The process of constructing the prompt is represented as:

[0020]

[0021] wherein, is the prompt template, is the feature set, is the constraint rule set.

[0022] As a further preferred scheme, in the foregoing method, the LoRA quantization technology adopts 4-bit quantization processing, and when the image is encoded into a visual embedding and mapped to the latent embedding space of the multi-modal large language model, layer normalization processing is performed through a linear projector to match the visual features with the embedding space distribution of the multi-modal large language model.

[0023] As a further preferred solution, in the aforementioned method, the fine-tuning framework comprises a visual transformer image encoder, a linear projector and a frozen pre-trained model.

[0024] Based on the second main aspect of the present application, a visual question answering (VQA) dataset construction method is provided, comprising the following steps:

[0025] The original image and associated annotation information are input into a computer system, preprocessed, divided into a training set and a test set according to a category balancing principle, and the feature images of each dataset sample are extracted and formatted to obtain an annotated dataset for a classification task.

[0026] The VQA dataset is constructed in the following two stages: in the first stage, the annotated dataset is converted into visual question answering (VQA) sample one; in the second stage, the observation report is converted into visual question answering (VQA) sample two; and the VQA samples of the two stages are fused to obtain a VQA dataset containing a multi-modal prompt template.

[0027] As a further preferred solution, in the aforementioned method, the visual question answering (VQA) sample one is expressed as follows:

[0028]

[0029] wherein, represents an instruction prompt associated with the image, forms a structured description tuple, which respectively contains visual representation, data source metadata and semantic analysis derived from the original label:

[0030] The visual question answering (VQA) sample two is expressed as follows:

[0031]

[0032] wherein represents an optical character recognition operator, represents artificial semantic annotation, corresponds to a predefined structured response function;

[0033] The mathematical expression of the VQA dataset containing a multi-modal prompt template is wherein the operator represents the fusion of semantically related samples.

[0034] Based on the third main aspect of the present application, an AI assistant system for implementing a multi-modal large model-based image analysis method is provided, comprising: a computer system deployed with a multi-modal large language model, and:

[0035] A data input module is configured to receive image analysis raw data, wherein the image analysis raw data comprises original images and associated annotation information.

[0036] A data preprocessing module is configured to preprocess the raw data, divide a training set and a test set according to a category balancing principle, extract feature images and unify formats, and output an annotation data set for a classification task.

[0037] A VQA prompt generation module is configured to convert the annotation data set for the classification task into visual question answering (VQA) prompts suitable for fine-tuning a multimodal large language model, including generating a basic question and answer pair based on a classification label, generating a deep question and answer pair in combination with a physical parameter, and constructing a structured prompt template.

[0038] A data set construction module is configured to integrate the VQA prompts into a VQA data set, perform classification annotation, divide the training set and the test set, and perform quality checking.

[0039] A model fine-tuning module is configured to perform knowledge injection and reasoning model optimization on the multimodal large language model based on the VQA data set, including constructing a fine-tuning framework using LoRA quantization technology, encoding images into visual embeddings and mapping them to a latent embedding space of the multimodal large language model, designing a multimodal prompt template, and adapting training parameters according to the model parameter scale.

[0040] A reasoning analysis module is configured to call the optimized multimodal large language model, fuse the image to be analyzed and the corresponding prompt template into a multimodal input, and perform multi-step logical deduction through the multimodal large language model to complete image analysis.

[0041] A result output module is configured to output the analysis results generated by the multimodal large language model, including image classification labels and visual question answering text answers.

[0042] According to a third main aspect of the present application, a computer readable storage medium is provided, which stores a computer program that, when executed, implements the image analysis method based on the multimodal large model according to any one of claims 1-6.

[0043] Compared with the prior art, the present application relies on the cross-task generalization ability of the multimodal large language model (MLLMs), combines a special VQA data set and an optimized reasoning mechanism, and can simultaneously complete the image classification and visual question answering dual tasks, with an analysis efficiency improved by more than 3 times compared with traditional methods, and without the need to repeatedly develop models for a single task, thereby reducing the operation cost of researchers.

[0044] At the level of analysis accuracy, the application solves the core problem that general MLLMs lack knowledge in the field of radio astronomy. Through the radio astronomy VQA dataset constructed by four types of open source radio astronomy data, professional knowledge such as pulsar dispersion measurement and radio galaxy morphological features can be injected into the model. Combined with LoRA quantitative fine-tuning and multi-modal prompt template design, the recognition accuracy of the model for pulsar candidates is significantly improved. The application can accurately output the professional logic question and answer results such as "radiation intensity peak corresponding frequency band" and "spectrum anomaly and solar activity association", and meet the strict requirements of radio astronomy research on accuracy.

[0045] The radio astronomy VQA dataset constructed by the application fills the gap in the field and provides a standardized basis for subsequent related technology research and development. Through the two-stage construction method of "classifying data into VQA prompts", the application forms a standardized dataset containing image-text association pairs and covering multiple types of celestial body samples, which not only meets the training needs of MLLMs, but also can be directly used by other radio astronomy intelligent analysis tools. The application can promote the sharing and reuse of data resources in the field and accelerate the landing process of intelligent technology in radio astronomy.

[0046] From the application value, the application reduces the technical threshold of radio astronomy data interpretation. The AI assistant platform of the application is designed through structured prompt templates and natural language interaction. Researchers do not need to deeply understand the principles of the model. They only need to input the image to be analyzed to obtain a structured report containing classification results, parameter analysis and reasoning basis. Non-AI professionals can also efficiently complete data interpretation, helping more research teams to mine scientific value from radio astronomy data. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description only some embodiments of the application, and for those skilled in the art, other drawings can be obtained according to these drawings without creative labor.

[0048] Figure 1 An execution flowchart of a radio astronomy image analysis method based on MLLMs in an embodiment of the application is shown. DETAILED DESCRIPTION

[0049] The preferred embodiments of the application will be described in detail below in order to more clearly understand the purpose, characteristics and advantages of the application. It should be understood that the following embodiments are not a limitation on the scope of the application, but only to illustrate the essential spirit of the technical solutions of the application.

[0050] In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of various disclosed embodiments. It will be appreciated, however, that embodiments can be practiced in other ways without one or more of the specific details set forth in this description. In other instances, well-known enclosures, structures, techniques, and applications are not re-described in detail in order to avoid obscuring this description.

[0051] Reference throughout this specification to "an embodiment" or "the embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0052] In the following embodiments, the professional terms that may be involved are explained as follows:

[0053] Radio astronomy image: The visualized image data formed by processing the radio waveband electromagnetic radiation signals emitted by celestial bodies (such as pulsars, radio galaxies, etc.) received by a radio telescope, which is the core observation data for studying the physical properties of celestial bodies in the universe.

[0054] MLLMs (Multimodal Large Language Models): Large-scale artificial intelligence models with the ability to process multiple modalities of data such as text and images, which can achieve cross-modal understanding, reasoning, and generation tasks through learning massive data, providing core algorithm support for the present invention.

[0055] VQA (Visual Question Answering): A technology task that combines computer vision and natural language processing, which requires the model to output logical natural language answers according to the input image and corresponding natural language questions, and is a key technology form for realizing image depth analysis in the present invention.

[0056] LoRA quantization technology (Low-Rank Adaptation Quantization): An efficient fine-tuning and compression technology for large language models, which reduces the amount of calculation and memory usage by introducing a low-rank matrix in the model parameters, while ensuring the fine-tuning effect, so that MLLMs can complete the knowledge injection in the field of radio astronomy under the conventional hardware environment.

[0057] Visual Embedding: A high-dimensional vector representation of image data transformed by a neural network model (such as an image encoder). It can transform the visual features of an image (such as celestial morphology, signal intensity distribution, etc.) into a numerical form that MLLMs can understand, thus achieving unified processing of image and text modalities.

[0058] The labeled dataset for classification tasks is a dataset formed by manually or automatically labeling raw radio astronomy images according to preset categories (such as "pulsar", "non-pulsar", "radio galaxy" etc.). It contains image data and corresponding category labels and is the basic data for subsequent conversion into VQA prompts and training models.

[0059] Multimodal prompt template: A structured input format that integrates visual tags (associating with image data), text queries (natural language problems), and task identifiers (distinguishing between task types such as classification and parameter parsing) to guide MLLMs to accurately focus on radio astronomy analysis tasks and improve the relevance and accuracy of model inference.

[0060] like Figure 1 As shown, in one embodiment of the present invention, a radio astronomy image analysis method based on a multimodal large model includes the following steps S110-S150:

[0061] S110 inputs raw data for radio astronomy image analysis into a computer system deployed with multimodal large language models (MLLMs);

[0062] S120: The computer system preprocesses the raw data, divides the training set and test set according to the class balance principle, extracts feature images and unifies the format to obtain the labeled dataset for the classification task.

[0063] S130 transforms the labeled dataset for classification tasks into visual question answering (VQA) tips suitable for fine-tuning MLLMs and constructs a radio astronomy VQA dataset.

[0064] S140, Based on the radio astronomy VQA dataset, knowledge injection and inference model optimization are performed on MLLMs, including: constructing a fine-tuning framework using LoRA quantization technology, encoding images into visual embeddings and mapping them to the latent embedding space of MLLMs, designing multimodal prompt templates that integrate visual tags, text queries and task identifiers, adapting training parameters according to the model parameter scale, enabling MLLMs to learn and structurally store knowledge in the field of radio astronomy, and optimizing their inference mechanism based on multimodal input;

[0065] S150, calling the MLLMs optimized by knowledge injection and reasoning, fusing the radio astronomy image to be analyzed and the corresponding prompt template into a multi-modal input, performing multi-step logical deduction through the learned domain knowledge and the optimized reasoning model of the MLLMs, completing image analysis, and outputting the analysis result generated by the reasoning of the MLLMs.

[0066] The data set collection and preprocessing are described in the following possible implementation.

[0067] In this embodiment, the original data sets for constructing the RadioAstroVQA data set and the preprocessing steps thereof. Specifically, these data sets come from the following four open source projects:

[0068] FAST data set: composed of 15,482 pulsar candidates collected by the Five-hundred-meter Aperture Spherical Telescope (FAST). This data set aims to solve the problem of how to distinguish real pulsars from radio frequency interference (RFI). Each sample contains a diagnostic report generated by the Presto system, which contains four chart images: profile, phase, subband, and dispersion measure (DM) images, with relevant search information attached. The data set is manually annotated by experts into two categories: pulsar and RFI, containing 1,163 known pulsars and 14,319 RFI instances. To solve the class imbalance problem, in this embodiment, 837 known pulsars and 1,593 RFI instances are randomly selected as the training set, and the remaining samples are all used as the test set.

[0069] HTRU Medlat data set: a collection of labeled pulsar candidates located in the medium galactic latitude region of the High Time Resolution Universe (HTRU) survey project. This data set was originally processed for training the SPINN pulsar classifier and contains 1,196 known pulsar candidates and 89,996 non-pulsar radio frequency interference (RFI) candidates from 521 different sources. Each sample contains three sets of visualization images: phase, subband, and velocity-momentum (DM) images. In this embodiment, 837 known pulsars and 1,593 RFI samples are randomly selected as the training set, and the remaining 359 known pulsars and 8,406 RFI samples are used as the test set.

[0070] Spectrumcls data set: contains 8,816 solar radio spectrum samples obtained by the Solar Broadband Radiometer (SBRS) of the National Astronomical Observatory of China. Each image in the data set is scanned at a resolution of 120 × 120 pixels and labeled as a radio burst, calibration, or non-burst type. In this embodiment, 7,052 samples are randomly selected as the training set, and the remaining 1,764 samples are used as the test set.

[0071] Radio Galaxy: contains 2158 radio galaxy sources from the FIRST survey, each image in the dataset is scanned at a resolution of 300 x 300 pixels and labeled as FRI, FRII, Compact, or Bent. In this embodiment, 1758 samples are randomly selected as the training set, and the remaining 400 samples are used as the test set.

[0072] In addition to the labeled images, the FAST dataset also uses the intermediate processing result analysis report provided by Presto for each sample; the HTRU Medlat dataset is based on the physical characteristic parameters proposed in existing research (F. Zhao, Y. Li, Y. Wang, H. Li, M. Chen, P. Chen, N. Sun, C. Wang, and J. Liu. Pulsar candidate classification with multimodal large language models. https: / / openreview.net / pdf?id=8SKgWpZiDL, 2024.) including the best period, the best dark current density, the best signal-to-noise ratio, and the pulse width. As for the Spectrumcls dataset, the subtitle information of all samples is directly used as the basis for analysis.

[0073] In the following possible implementation, how to convert the labeled dataset of the classification task into a visual question answering (VQA) prompt suitable for fine-tuning multimodal large language models (MLLMs) will be described.

[0074] Existing deep learning methods for radio astronomy tasks can usually only process images or physical features one at a time. However, actual datasets often contain multiple types of images and their corresponding physical features, so the generated prompts should be flexible enough to accept one or more images and other numerical and categorical attributes as input. In this way, there is no need to redesign the model for each type of chart provided by different projects in this embodiment.

[0075] As a general process that can handle multiple input types, in this embodiment, the image input Three modes are designed: single, multiple, and combination. In single mode, this embodiment only uses a single frame of image as input, and this mode is only enabled when a single frame of image is used.

[0076] In multi-mode, all available images are input into the visual encoder one by one in the embodiment, and the features are spliced to complete the projection. In the combination mode, all available images are integrated into a single input image into the visual encoder. In addition, in the embodiment, numerical features and related heuristics are embedded in the prompt template. The feature extraction in this process can effectively capture the physical characteristics of specific radio signals. However, not all features are equally important for prediction performance.

[0077] In most embodiments, existing mature feature selection techniques based on various machine learning algorithms, including logistic regression, ridge regression, random forest, support vector machine (SVM), LightGBM and CatBoost, can be used to evaluate the importance of these features. Through a majority voting mechanism, the optimal feature set for the MLM input is determined in the embodiment.

[0078] At the same time, in some embodiments, heuristics based on expert experience are also used to effectively integrate domain knowledge into task processing. Overall, the process of building prompts can be represented as:

[0079]

[0080] wherein, is the prompt template, is the feature set, is the constraint rule set.

[0081] The construction of the VQA dataset is described in the following possible implementation.

[0082] Unlike downstream classification tasks, the visual question answering (VQA) task does not rely on a single data source, but requires a more comprehensive approach to build a model that understands images. In the field of radio astronomy, observation reports are often not uniform in format and are scattered in multiple important survey data sources. There are limited publicly labeled datasets, which poses a major challenge to the construction of the VQA dataset.

[0083] Therefore, in the embodiment, a two-stage strategy is adopted to construct the VQA dataset:

[0084] First, four publicly labeled image datasets, FAST, HTRU Medlat, Spectrumcls and radio Galaxy, are integrated to ensure diversity of data sources; second, based on the FAST and Spectrumcls datasets, result files are generated through manual analysis or special software to supplement key parameter information in self-built reports.

[0085] Next, the two stages of VQA dataset construction will be described in detail in the embodiment.

[0086] Phase 1: Transform the annotated dataset into visual question answering (VQA) samples. Standard visual datasets usually contain sample instances where and are generated from the flow of constructing prompts according to the previous step, each sample with a corresponding category label . These samples collectively constitute the basis of the VQA dataset, labeled as . From the formal description, the basic data format can be represented as:

[0087]

[0088] where the index uniquely identifies each data instance, represents the sample index set.

[0089] In this embodiment, the basic information is expanded by integrating image descriptions, data sources, and brief analyses based on labels to construct relevant prompts. Taking a sample from the FAST dataset as an example, this embodiment uses the prompt "Please analyze the following pulsar candidate image." to describe the task requirements. Then, the MLM model is asked to give an answer <image analysis>, which needs to include: i) image description (e.g., "This image is a diagnostic grayscale image of a pulsar candidate, possibly a time-phase or frequency-phase image."); ii) data source (e.g., "It may be generated by pulsar search software such as PRESTO."); iii) category label and brief analysis (e.g., "The signal is in a disordered state, possibly a radio frequency interference (RFI) signal."). The processing of samples from other annotated datasets is similar. The detailed prompt templates for each dataset are shown in the table. In summary, after converting all samples from the four annotated datasets, the first phase visual question answering dataset is expressed as follows:

[0090]

[0091] where represents the instruction prompt associated with the image, constitutes a structured description tuple, containing visual representation, data source metadata, and semantic analysis derived from the original label. This structured conversion framework can effectively convert the original data into interactive question and answer pairs, significantly improving the semantic richness and practical application value.

[0092] Phase 2: Transform the observation report into visual question answering (VQA) samples. Although serves mainly for downstream classification tasks, its representation dimension is limited, but some original data (such as spectral classification reports) fully retain the context semantics, reflecting the multi-modal correlation between observation text and corresponding image regions.

[0093] The mixed information extraction framework of the present application comprises two core steps: (1) extracting text from predefined image regions by a document intelligence tool named Doc2X; (2) combining human-annotated to capture potential semantic attributes (e.g., specific timestamps of solar flare events). The synthesized features are then passed through a response template function implementation.

[0094] Therefore, the second stage of VQA dataset construction is

[0095]

[0096] where denotes an optical character recognition operator, denotes human semantic annotation, corresponds to a predefined structured response function;

[0097] The mathematical expression of the radio astronomy VQA dataset is where the operator denotes the fusion of semantically relevant samples.

[0098] The following possible implementation explains the fine-tuning of MLLMs for radio astronomy tasks.

[0099] To optimize the performance of multilingual language models (MLLMs) in radio astronomy tasks, the present embodiment fine-tunes pre-trained MLLMs models using the RadioAstro VQA dataset to achieve image classification and visual quality assessment (VQA) tasks.

[0100] To improve the computational efficiency of the fine-tuning process, the present embodiment employs quantization techniques and LoRA technology, which is a parameter optimization scheme based on low-rank matrix decomposition. The fine-tuning framework comprises three core modules: i) a visual transformer (ViT) based image encoder for generating visual embeddings; ii) a linear projector that maps visual features to a fixed 4096-dimensional latent embedding space of LLM; iii) a frozen pre-trained LLM model (optional LoRA optimization). Memory optimization is achieved by tokenizing serialized image features and concatenating along the embedding dimension. The visual encoder parameters are continuously learned during the domain adaptation process.

[0101] In the present embodiment, an integrated multi-modal prompt template is also designed, which integrates image features, text queries, and task identifiers (e.g., ([VQA])). The format specification of this task is "ImageFeature: [text query] Task: [VQA]" ImageFeature[TaskID]". The dataset in this example contains input prompts for radio astronomy tasks and their corresponding standard output results (labeled I and O, respectively). The generation probability is calculated using the following formula:

[0102]

[0103] in Indicates the first The autoregressive probability of each step. The fine-tuning objective is to minimize the following loss function:

[0104]

[0105] in This represents the dataset used for fine-tuning. Therefore, the fine-tuning framework effectively captures the relationship between input and output, laying a solid foundation for optimizing the performance of multi-head language models in broadcast astronomy missions.

[0106] During fine-tuning, this embodiment uses two open-source multilevel linear models (MLLMs) with different parameter scales as the base models: DeepSeek-VL-7B and InternVL2-40B. The default optimizer is AdamW, and training stability is ensured by setting the maximum norm of gradient clipping to 1. A cosine learning rate scheduling function is used to achieve smooth convergence, along with a 5% warm-up ratio.

[0107] The AI ​​assistant system of the present invention will be described in one of the following possible implementations.

[0108] In this embodiment, the hardware foundation of the AI ​​assistant system is a high-performance computer system that deploys multimodal large language models (MLLMs). The system is equipped with a GPU acceleration cluster to support model training and inference. The MLLMs can be pre-trained models with visual-language understanding capabilities, such as LLaVA and GPT-4V.

[0109] In this embodiment, the data input module receives four types of open-source radio astronomy data through a standardized interface, including raw survey images in FITS format (such as SKA simulation data and NVSS survey data), target source annotation files (including right ascension / declination coordinates), spectral energy distribution data, and astrophysical parameter tables (such as redshift and flux density), supporting batch uploading and real-time streaming data access.

[0110] The data preprocessing module uses the following process to handle raw data. First, based on the celestial body category label (such as quasar, galaxy cluster, supernova remnant), the sample is balanced, and the SMOTE algorithm is used to generate synthetic samples for the few categories. Second, the target source area in the image is extracted through the ASTROPY library, and the feature image is cropped to 224x224 pixels and converted to PNG format. Finally, the processed image and the corresponding annotation information (category label, physical parameter) are associated to generate a CSV format classification task dataset, which is randomly divided into training set and test set in the ratio of 7:3.

[0111] In this embodiment, the VQA prompt generation module constructs a two-level question and answer system. The basic question and answer pair is for image classification tasks, such as "Which celestial body in the image belongs to which category? Answer: quasar." The deep question and answer pair combines physical parameters, such as "If the redshift value of this celestial body is 0.5, what is its corresponding distance? Answer: 650 million light years."

[0112] At the same time, a structured prompt template is designed, including image description ("The following is a celestial body image taken in the radio wave band"), question type ("Please determine the type of celestial body and calculate its physical parameters"), and output format constraint ("Return the category and parameter value in JSON format"). The dataset construction module integrates the above VQA prompts into the radio astronomy VQA dataset, removes ambiguous samples through manual verification, and finally forms a standardized dataset containing 100,000 training samples and 30,000 test samples.

[0113] In this embodiment, the model fine-tuning module uses LoRA quantization technology to build a lightweight fine-tuning framework. In the case of keeping the main parameters of MLLMs frozen, only the low-rank adaptive matrix is trained to reduce the consumption of computing resources. The specific process is as follows: first, use the CLIP visual encoder to encode the preprocessed image into a 768-dimensional visual embedding, and map it to the word embedding space of MLLMs (such as 4096-dimensional space for 7B parameter model) through a linear projection layer. The mapped visual embedding and the text prompt are concatenated into a multi-modal input, and a causal language model loss function is used to optimize the model. Training parameters are adapted for different sizes of models, such as batch size=16, learning rate=2e-4 for 7B model, batch size=8, learning rate=1e-4 for 33B model, and training period is set to 10 rounds.

[0114] After receiving the radio astronomy image to be analyzed, the reasoning analysis module automatically calls the pre-processing procedure to generate a feature image, selects a matching prompt template according to the task type (classification / parameter calculation), and forms a multi-modal input of image embedding and text prompt. The MLLMs complete the analysis through multi-step logical deduction, for example, first identifying the morphological features in the image (“there is a double-lobe structure”), then combining the training knowledge to infer the category (“it may be a radio galaxy”), and finally calculating the derived quantity according to the physical parameter formula (“calculate the luminosity according to the flux density”).

[0115] The result output module presents the analysis results in dual form: structured data (including classification labels, confidence, and physical parameter values) for subsequent data processing, and natural language text (such as “the celestial body is a quasar with a redshift of 0.3 and a luminosity of 1e45 erg / s”) for intuitive display, supporting JSON, PDF, and other formats for export.

[0116] In the above embodiments, technical terms, technical principles, or technical means related to the technical solutions of the present application are involved, and those not described in detail in the above content are well-known to those skilled in the art or common means.

[0117] The basic principles and main features of the present application and the advantages of the present application have been shown and described above. It should be understood by those skilled in the art that the present application is not limited by the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A multi-modal large model-based image analysis method, characterized by, The method comprises the following steps: inputting image analysis raw data into a computer system deployed with a multimodal large language model; the computer system pre-processes the raw data, divides the training set and the test set according to the category balance principle, extracts the feature image and unifies the format, and obtains the labeled data set of the classification task; the labeled data set of the classification task is converted into a visual question answering (VQA) prompt suitable for fine-tuning the multimodal large language model, and a VQA data set is constructed; based on the VQA data set, the multimodal large language model is subjected to knowledge injection and reasoning model optimization, including: using the LoRA quantization technology to construct a fine-tuning framework, encoding the image into a visual embedding and mapping it to the latent embedding space of the multimodal large language model, designing a multimodal prompt template integrating visual markers, text queries and task identifiers, adapting the training parameters according to the model parameter size, making the multimodal large language model learn and structure the field knowledge, and optimizing its reasoning mechanism based on multimodal input; the multimodal large language model subjected to knowledge injection and reasoning optimization is called to fuse the image to be analyzed and the corresponding prompt template into a multimodal input, and through the multimodal large language model, the learned field knowledge and the optimized reasoning model are called to perform multi-step logical deduction, complete image analysis, and output the analysis result reasoned and generated by the multimodal large language model. 2.The multi-modal large model based image analysis method according to claim 1, wherein, The image analysis raw data includes four types of open source radio astronomy raw images and associated labeled information, and the labeled information includes astrophysical parameters and observation feature descriptions. The four types of open source radio astronomy raw data include: pulsar candidate images and labeled tags of the FAST data set, pulsar candidate images and physical property parameters of the HTRU Medlat data set, solar radio spectrum sample images and subtitle information of the Spectrumcls data set, and radio galaxy source images and morphological annotations of the Radio Galaxy data set. 3.The multi-modal large model based image analysis method of claim 1, wherein, When pre-processing the raw data, the feature images include: extracting profile graphs, phase graphs, sub-band graphs and dispersion measurement graphs from the FAST data set samples; extracting three phase graphs, sub-band graphs and velocity-momentum graphs from the HTRU Medlat data set samples; and uniformly adjusting the Spectrumcls data set sample images to a preset resolution. 4.The multi-modal large model based image analysis method of claim 1, wherein, The labeled data set of the classification task is converted into a VQA prompt suitable for fine-tuning the multimodal large language model, including: image input is designed in three modes: single, multiple and combination; in the single mode, only a single frame image is input; in the multiple mode, all available images are input into the visual encoder one by one, and the features are spliced to complete projection; in the combination mode, all available images are integrated into a single input image and input into the visual encoder; numerical features and empirical rules are embedded in the prompt template to determine the optimal feature set for multimodal large language model input and integrate field knowledge into task processing; the process representation of the prompt is: ； wherein, is a prompt template, is a feature set, is a constraint rule set. 5.The multi-modal large model based image analysis method according to claim 1, wherein, The LoRA quantization technology adopts 4-bit quantization processing. When the image is encoded into a visual embedding and mapped to a latent embedding space of a multi-modal large language model, layer normalization is performed by a linear projector to match the visual features with the embedding space distribution of the multi-modal large language model.

6. The multimodal large model-based image analysis method according to claim 5, wherein, The fine-tuning framework includes a visual transformer image encoder, a linear projector, and a frozen pre-trained model. 7.A method for constructing a visual question answering (VQA) dataset, comprising: The method includes the following steps: The original image and associated annotation information are input into the computer system, the original image and associated annotation information are preprocessed, the training set and test set are divided according to the category balance principle, the feature images of the samples in each dataset are extracted and unified in format, and the annotation dataset for the classification task is obtained; The VQA dataset is constructed in the following two stages: in the first stage, the annotation dataset is converted into visual question answering (VQA) sample one; in the second stage, the observation report is converted into VQA sample two; and the VQA samples of the two stages are fused to obtain a VQA dataset containing a multi-modal prompt template.

8. The method of claim 7, wherein, The VQA sample one is expressed as follows: ； wherein, represents an instruction cue associated with the image, constitute a structured description tuple, containing visual representation, data source metadata and semantic analysis derived from the original label, respectively: The VQA sample two is expressed as follows: ； wherein represents an optical character recognition operator, represents a human semantic annotation, corresponds to a predefined structured response function; The mathematical expression of the VQA dataset comprising the multi-modal prompt template is where the operator denotes the fusion of the semantically relevant samples. 9.An AI assistant system implementing a multi-modal large model-based image analysis method, characterized by, The method includes the following: A computer system deployed with a multi-modal large language model, and A data input module for receiving image analysis raw data, the image analysis raw data including an original image and associated annotation information; A data preprocessing module for preprocessing the raw data, dividing the training set and test set according to the category balance principle, extracting the feature images and unifying the format, and outputting the annotation dataset for the classification task; A VQA prompt generation module for converting the annotation dataset for the classification task into a visual question answering (VQA) prompt suitable for fine-tuning the multi-modal large language model, including generating a basic question and answer pair based on the classification label, generating a deep question and answer pair combined with physical parameters, and constructing a structured prompt template; A dataset construction module for integrating the VQA prompt into a VQA dataset, performing classification annotation, dividing the training set and test set, and quality checking; A model fine-tuning module for knowledge injection and reasoning model optimization of the multi-modal large language model based on the VQA dataset, including constructing a fine-tuning framework using LoRA quantization technology, encoding the image into a visual embedding and mapping it to a latent embedding space of the multi-modal large language model, designing a multi-modal prompt template, and adapting the training parameters according to the model parameter size; A reasoning analysis module for calling the optimized multi-modal large language model, fusing the image to be analyzed and the corresponding prompt template into a multi-modal input, and performing multi-step logical deduction through the multi-modal large language model to complete the image analysis; A result output module for outputting the analysis result generated by the multi-modal large language model, including the image classification label and the visual question answering text answer.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed to implement the multi-modal large model-based image analysis method of any one of claims 1-6.

Images