Meteorological service multi-modal large model construction method and device

Abstract

The invention discloses a meteorological service multi-mode large model construction method and device, and belongs to the field of artificial intelligence. According to the method, firstly, through field adaptive increment pre-training on meteorological service data, a model obtains a preliminary visual understanding capability facing a meteorological service scene; and then, a multi-stage course learning strategy combining instruction alignment and preference alignment is adopted to guide the model to further enhance the instruction following ability and task generalization on the basis of the field-specific visual understanding ability, so that the adaptability of the model to various meteorological service tasks is improved. The trained visual big language model can adapt to professional image understanding tasks in various meteorological service scenes, has the capability of generating high-quality professional text description according to image content under a given instruction, and effectively meets the actual requirements of applications such as weather forecast service and meteorological early warning service.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method and apparatus for constructing a multimodal large model of meteorological services. Background Technology

[0002] Vision-Language Models (VLMs) have demonstrated excellent perception and cognition capabilities in general vision tasks. However, their practical application still faces significant technical bottlenecks in long-tail application scenarios such as meteorological services, which are highly specialized and have highly privatized data.

[0003] Meteorological service tasks typically involve multimodal data processing, such as generating accurate text descriptions of weather conditions based on professional images like weather forecast maps and meteorological disaster warning maps. This process requires the integration of precise geospatial positioning and comprehensive identification of multiple meteorological elements, making the task highly complex and making it difficult for general-purpose visual language models to directly meet actual business needs. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method and apparatus for constructing a multimodal large model for meteorological services. The trained visual large language model can adapt to professional image understanding tasks in various meteorological service scenarios and has the ability to generate high-quality professional text descriptions based on image content under given instructions, effectively meeting the practical needs of applications such as weather forecasting services and meteorological early warning services.

[0005] The technical solution provided by this invention is as follows:

[0006] A method for constructing a multimodal large model for meteorological services, the method comprising:

[0007] S1: Build a large visual language model and initialize it using the pre-trained model parameters on a general visual language dataset;

[0008] The visual large language model includes a visual encoder, a visual-language adapter, and a large language model.

[0009] S2: Freeze the visual encoder and large language model, and use the constructed meteorological service domain image-text pair dataset to incrementally pre-train the visual-language adapter, so that the visual-language adapter learns the specific semantic features and expressions of meteorological service images;

[0010] S3: End the freeze on the visual encoder and the large language model, and use the constructed instruction alignment dataset containing large-scale, diverse meteorological service image-text pairs to train the visual large language model in a full-parameter model mode for instruction alignment, so as to enhance the visual large language model's basic ability to follow meteorological service instructions.

[0011] S4: Freeze the visual encoder and visual-language adapter, and use a preference dataset containing rare key meteorological events to train the visual large language model with low-rank adaptive parameters, thereby enhancing the generalization ability of the visual large language model in meteorological service scenarios.

[0012] Furthermore, for the images used in training, the images are divided according to resolution and aspect ratio. Each of the following local image tiles has a resolution of [resolution value missing]. And scale the original image to The thumbnail is used to stitch together all local image tiles and the thumbnail into a visual input, which is then fed into a visual encoder to extract visual features.

[0013] Local image tiles are used to capture fine-grained regional information, while thumbnails are used to provide global semantic context.

[0014] Furthermore, during incremental pre-training of the vision-language adapter, class-balanced random sampling is used for all meteorological service domain image-text pair datasets to ensure the diversity of training data distribution.

[0015] Furthermore, when training the visual large language model with full parameter alignment, the instruction alignment dataset is divided into multiple ordered training stage datasets from simple to complex according to the complexity of the text data and image data. The visual large language model performs sequential learning from simple to complex according to the order of the ordered training stage datasets.

[0016] Furthermore, for text data, the complexity is categorized based on text length and professional complexity; for image data, the complexity is categorized based on the clarity of image background and meteorological elements; and for supplementary data of new scenarios and new types, they are directly assigned to the most complex ordered training stage dataset.

[0017] Furthermore, when training the visual large language model for preference alignment, the following loss function is used. ;

[0018]

[0019] in, To optimize loss for direct preference, To monitor and fine-tune losses, The set coefficient.

[0020] A device for constructing a multimodal large model of meteorological services, the device comprising:

[0021] The model building module is used to build a visual large language model and initialize it using the pre-trained model parameters on a general visual language dataset;

[0022] The visual large language model includes a visual encoder, a visual-language adapter, and a large language model.

[0023] The incremental pre-training module is used to freeze the visual encoder and the large language model. It uses the constructed meteorological service domain image-text pair dataset to perform incremental pre-training on the visual-language adapter, enabling the visual-language adapter to learn the specific semantic features and expressions of meteorological service images.

[0024] The instruction alignment module is used to end the freezing of the visual encoder and the large language model. It uses a fully parameterized model training method to perform instruction alignment on the visual large language model using an instruction alignment dataset containing large-scale and diverse image-text pairs in the meteorological service domain, thereby enhancing the visual large language model's basic ability to follow meteorological service instructions.

[0025] The preference alignment module is used to freeze the visual encoder and visual-language adapter. It uses a constructed preference dataset containing rare key meteorological events to train the visual large language model with low-rank adaptive parameters, thereby enhancing the generalization ability of the visual large language model in meteorological service scenarios.

[0026] Furthermore, for the images used in training, the images are divided according to resolution and aspect ratio. Each of the following local image tiles has a resolution of [resolution value missing]. And scale the original image to The thumbnail is used to stitch together all local image tiles and the thumbnail into a visual input, which is then fed into a visual encoder to extract visual features.

[0027] Local image tiles are used to capture fine-grained regional information, while thumbnails are used to provide global semantic context.

[0028] Furthermore, during incremental pre-training of the vision-language adapter, class-balanced random sampling is used for all meteorological service domain image-text pair datasets to ensure the diversity of training data distribution.

[0029] Furthermore, when training the visual large language model with full parameter alignment, the instruction alignment dataset is divided into multiple ordered training stage datasets from simple to complex according to the complexity of the text data and image data. The visual large language model performs sequential learning from simple to complex according to the order of the ordered training stage datasets.

[0030] Furthermore, for text data, the complexity is categorized based on text length and professional complexity; for image data, the complexity is categorized based on the clarity of image background and meteorological elements; and for supplementary data of new scenarios and new types, they are directly assigned to the most complex ordered training stage dataset.

[0031] Furthermore, when training the visual large language model for preference alignment, the following loss function is used. ;

[0032]

[0033] in, To optimize loss for direct preference, To monitor and fine-tune losses, The set coefficient.

[0034] The present invention has the following beneficial effects:

[0035] The visual language model training strategy proposed in this invention is optimized for typical tasks in meteorological services, which helps improve the model's adaptability and accuracy in diverse application scenarios, thereby providing a more comprehensive and precise intelligent solution for the meteorological service field. The trained visual language model can adapt to professional image understanding tasks in various meteorological service scenarios and has the ability to generate high-quality professional text descriptions based on image content under given instructions, effectively meeting the practical needs of applications such as weather forecasting and meteorological early warning services. Furthermore, this invention has good scalability, facilitating continuous model optimization during actual deployment and flexibly adapting to various segmented business scenarios. Attached Figure Description

[0036] Figure 1 The flowchart is a method for constructing a multimodal large model for meteorological services according to the present invention.

[0037] Figure 2 This is a diagram illustrating the overall architecture of the visual large language model constructed in this invention.

[0038] Figure 3 This is a schematic diagram illustrating the overall process of the pre-training and post-training phases.

[0039] Figure 4 This is a schematic diagram of the meteorological service multimodal large model construction device of the present invention. Detailed Implementation

[0041] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0042] Example 1:

[0043] This invention provides a method for constructing a multimodal large model for meteorological services, such as... Figure 1 As shown, the method includes:

[0044] S1: Build a visual large language model and initialize it using the pre-trained model parameters on a general visual language dataset.

[0045] The visual large language model includes a visual encoder (ViT), a visual-language adapter (Adaptor), and a large language model (LLM).

[0046] The visual encoder and the large language model initialized the self-pre-trained InternViT-300M and Qwen3-14B models, respectively. The visual-language adapter was constructed by... The model is constructed using a layered perceptron (MLP), and the parameters of the vision-language adapter are initialized from a vision-language model pre-trained on a large-scale general vision-language dataset. The constructed large-scale vision-language model architecture follows the "ViT-MLP-LLM" structural paradigm, and the overall model architecture is as follows: Figure 2 As shown.

[0047] S2: Freeze the visual encoder and large language model, and use the constructed meteorological service domain image-text pair dataset to incrementally pre-train the visual-language adapter, enabling the visual-language adapter to learn the specific semantic features and expressions of meteorological service images.

[0048] This stage is the pre-training stage. To enhance the visual language model's ability in image semantic modeling, this stage focuses on training the visual-language adapter, enabling it to align visual features to the input space of the language model. This training strategy freezes both the visual encoder and the large language model, optimizing only the visual-language adapter implemented by a multilayer perceptron (MLP) to establish an efficient cross-modal mapping between them.

[0049] The vision-language adapter is first initialized using pre-trained parameters on a general vision-language dataset to fully utilize the existing capabilities of the general VLM, providing a good starting point for cross-modal alignment of the model and significantly improving training stability.

[0050] Subsequently, the visual-language adapter was incrementally pre-trained using a high-quality image-text pair dataset from the meteorological service domain. This enabled domain adaptation and cross-modal semantic alignment between images and text within the meteorological service domain. The model gradually mastered the specific semantic features and expressions of meteorological service images, thus endowing it with domain-specific visual understanding capabilities for meteorological services. During training, a class-balanced random sampling strategy was employed across all meteorological service datasets to mitigate bias caused by class imbalance and ensure coverage and diversity of different meteorological elements and scenes during training, effectively improving the model's generalization ability across various meteorological service tasks.

[0051] During the incremental pre-training phase, the visual language model is formalized as follows: , in Represents a visual encoder. Indicates a visual-language adapter. Represents a large language model. Visual-language adapter initialization parameters. It was obtained after pre-training on a massive general visual language dataset.

[0052] Based on this, incremental pre-training is conducted. This applies to text-image data in the meteorological service field. The training data is obtained through class-balanced sampling.

[0053]

[0054] This is to ensure that the model can generalize well across diverse categories of meteorological service images.

[0055] The objective function for incremental pre-training is:

[0056]

[0057] in Represents cross-entropy loss, Represents meteorological service images. This represents the corresponding text description.

[0058] S3: End the freeze on the visual encoder and the large language model, and use the constructed instruction alignment dataset containing large-scale, diverse meteorological service image-text pairs to train the visual large language model in a full-parameter model mode for instruction alignment, thereby enhancing the visual large language model's basic ability to follow meteorological service instructions.

[0059] This step, along with the subsequent S4, constitutes the post-training phase following pre-training. The post-training phase employs a multi-stage course learning strategy to progressively improve the model's instruction compliance ability in meteorological service tasks. The entire process consists of two core stages: instruction alignment (this step) and preference alignment (S4). The overall process of the pre-training and post-training phases is as follows: Figure 3 As shown.

[0060] During the instruction alignment phase, a pre-trained visual language model is fine-tuned under supervision using large-scale and diverse image-text pairs provided by meteorological services. This process aims to enhance the model's ability to understand and execute domain-specific instructions, thereby forming a foundational meteorological service visual language model with robust instruction compliance capabilities tailored to meteorological service tasks.

[0061] In visual large language models, the instruction alignment process is formalized as a supervised learning task. Given an instruction alignment dataset...

[0062]

[0063] in, This indicates the input of professional images in the meteorological field. Representing natural language instructions. This represents the corresponding target response. (Model) Trained to generate output sequences ,in These are the model parameters that undergo domain adaptation through incremental pre-training, and this output needs to be consistent with the input instructions. Its learning objective is to minimize the cross-entropy loss between the predicted sequence and the true sequence.

[0064]

[0065] in, Indicates the location The target token, This represents a previously generated token. A full-parameter fine-tuning strategy is used to optimize the model parameters. This is to ensure that the VLM can correctly understand and follow input instructions under given visual context.

[0066] Due to the diverse and significantly complex nature of meteorological service scenarios, the entire instruction alignment dataset... One-off, homogeneous fine-tuning is unlikely to achieve optimal results. Therefore, this invention structures the training data and gradually adjusts the data distribution and task complexity during training. This strategy draws inspiration from the human learning pattern of "from easy to difficult," aiming to improve the model's training efficiency and final performance.

[0067] That is, when training the visual large language model with full parameter alignment, the instruction alignment dataset is divided into multiple ordered training stage datasets from simple to complex according to the complexity of the text data and image data. The visual large language model performs sequential learning from simple to complex according to the order of the ordered training stage datasets.

[0068] Specifically, this invention aligns the instruction dataset from both text and image dimensions. Conduct difficulty assessment and grading:

[0069] 1) Text Dimension: Difficulty is categorized by text length and technical complexity. Simple Samples: Contain concise and intuitive texts describing common weather phenomena. Complex Samples: Contain long sentences, technical terms, or descriptions of rare and severe weather events such as flood season or severe convection.

[0070] 2) Image Dimension: Difficulty is categorized based on the clarity of the image background and meteorological elements. Simple samples: Concise background, single and prominent meteorological features, high signal-to-noise ratio, conducive to model learning. Complex samples: Complex background or multiple meteorological phenomena superimposed, containing more interference information, requiring higher recognition capabilities from the model.

[0071] Based on the above definition of difficulty, the entire dataset will be aligned. Divided into multiple ordered training phase datasets ,satisfy: .

[0072] initial stage The initial training dataset consists primarily of simple samples, with more complex samples gradually introduced in subsequent stages. Simultaneously, supplementary data for new scenarios and types are directly incorporated into the more complex, ordered training dataset. During the post-training phase, the visual language model sequentially learns according to this order, first mastering core fundamental concepts and then gradually generalizing to complex, segmented business scenarios, ultimately achieving continuous enhancement in instruction alignment capabilities and task adaptability. Finally, the model is optimized into a visual language instruction model for meteorological service tasks, possessing strong instruction compliance and task execution capabilities, supporting various application scenarios such as weather forecasting services, disaster monitoring, and meteorological early warning.

[0073] S4: Freeze the visual encoder and visual-language adapter, and use a preference dataset containing rare key meteorological events to train the visual large language model with low-rank adaptive parameters, thereby enhancing the generalization ability of the visual large language model in meteorological service scenarios.

[0074] This step is used for preference alignment. Building upon instruction alignment, this stage focuses on improving the generalization ability of the basic meteorological service visual language model in long-tailed meteorological service scenarios. Rare but critical events, such as "heavy rainfall in Xinjiang or Qinghai," heat waves on the Qinghai-Tibet Plateau, and "persistent strong winds in South China," are often underrepresented in the training data, leading to data bias and degraded model performance. To address this issue, a preference dataset is constructed by automatically identifying low-frequency but valuable samples using keyword filtering based on regular expressions. Preference alignment training is then applied to guide the model to strengthen its response to these marginal cases, mitigating data distribution bias and enhancing the model's generalization ability and robustness in long-tailed meteorological scenarios.

[0075] The post-training phase employs a multi-stage course learning strategy (instruction alignment and preference alignment) to continuously enhance the model's instruction compliance and task execution capabilities in different meteorological service scenarios while gradually improving data quality (covering data scale, type diversity, and complexity). This enables the visual big language model to have strong instruction compliance capabilities and better generalization in complex meteorological service scenarios.

[0076] Direct Preference Optimization (DPO) is an alignment method that directly fine-tunes a language model based on human preference data. It transforms preference learning into a simple classification task through mathematical derivation, thus bypassing the complex processes of training reward models and reinforcement learning in traditional human-feedback-based reinforcement learning (RLHF). DPO directly utilizes preference pairs and employs a contrastive learning objective to constrain the relationship between the model's generated distribution and the reference model's distribution, achieving a more stable and efficient optimization process.

[0077] Given a preference dataset, where each sample contains two outputs: These represent preferred and dispreferred candidates, respectively.

[0078] Let the model parameters be The reference model parameters are The optimization objective of DPO is defined as follows:

[0079] The DPO (Direct Preference Optimization) algorithm optimizes directly based on human preferences, avoiding the training of an explicit reward model in the reinforcement learning phase. Its objective function is defined as follows:

[0080]

[0081] This expression represents the model parameters. In the input Below, samples of human preferences The learning process. Its core idea is: if the model outputs the "optimal" value... The relative probability of the "unfavorable selection" is higher than that of the reference model, while the output of the "unfavorable selection" is higher. If the relative probability is lower, then the loss function This will decrease, thereby encouraging models to generate results that are more in line with human preferences.

[0082] in DPO loss function; : For input samples and their corresponding preference pairs Expectations; The current model to be optimized is based on the input... Output generated below The conditional probability; The reference model (the SFT model built during the instruction alignment stage) is at the input... Output generated below The conditional probability; Humans prefer the "better" candidate output in the sample. : Poor candidate outputs from human preference samples; Temperature coefficient, used to control the strength of preference alignment; The Sigmoid function maps the logarithmic probability difference to... Intervals are used to smooth the gradient.

[0083] This objective function embodies the core concept of DPO: to achieve direct alignment of the model with human preferences through the relative comparison of probability distributions, without rewarding the model or using a reinforcement learning process.

[0084] DPO directly utilizes preference pairs in the data, employing a contrastive learning objective to constrain the relationship between the model-generated distribution and the reference model distribution, achieving a more stable and efficient optimization process. However, in specific domains such as meteorological services, the differences between positive and negative samples in the constructed preference pairs are minimal, making it difficult for the model to effectively distinguish them, potentially leading to instability in pure DPO training. To improve training robustness, this invention introduces Supervised Fine-Tuning (SFT) loss into the DPO loss function, constraining model behavior through joint optimization, thereby effectively mitigating the aforementioned problems.

[0085]

[0086] in, To optimize loss for direct preference, To monitor the fine-tuning (SFT) loss, The set coefficient.

[0087] The following demonstrates how the combined loss ensures training stability in specific meteorological service scenarios:

[0088]

[0089] And for a single-sample DPO, there is

[0090]

[0091]

[0092] Bounded gradient has a constant. Makes any ;

[0093]

[0094] Smoothness has a constant. , making In the area of ​​interest Upgradient Lipschitz:

[0095]

[0096] The merged gradients are bounded and there exists a constant. The instantaneous gradient for any sample (single-sample stochastic gradient estimation) satisfies

[0097]

[0098] The key points of the proof are directly expanded and the gradients are merged.

[0099]

[0100] and use and That is, the boundary is obtained.

[0101] Order in the The stochastic gradient estimate of step is And assume it is unbiased:

[0102]

[0103] The single-step expectation descent inequality is given below (standard smooth function + SGD analysis with noisy gradient).

[0104] Single-step expectation descent, let the gradient step size satisfy . but

[0105]

[0106] The last term on the right is introduced by the stochastic gradient variance.

[0107] The key to the proof lies in the smoothness.

[0108]

[0109] Substitution And take the expectation of the condition ( The above formula can be obtained by rearranging the equation (standard SGD derivation).

[0110] As can be seen from the theorem: as long as the step size satisfies Then a negative quadratic term dominates, thus affecting the error term. Under controllable conditions, expected losses will decrease or tend to stabilize.

[0111] The effect of SFT on stability (variance and direction)

[0112] Boundedness: Lemma guarantees an upper bound on the single-step gradient norm. ,thereby The upper bound of variance can be controlled by cropping / scaling.

[0113] Variance reduction: The actual stochastic gradient can be decomposed into .like For deterministic (or low variance) signals and with If there is positive consistency in direction, then the overall variance... It will have a lower variance than DPO alone. In engineering, this can be addressed by selecting appropriate... Achieving a trade-off between variance and bias.

[0114] Small bias case: When the difference between preference pairs is very small, the coefficient of DPO is... When smaller, the SFT item still provides information about The "absolute" signal is used to avoid ambiguity in optimization direction or being dominated by noise.

[0115] in conclusion:

[0116] Under the common assumptions of bounded gradient and smoothness, the combined loss

[0117]

[0118] It is stable under small step size SGD: there is a suitable upper bound on the learning rate that allows the expected loss to decrease step by step (to the stationary point), and SFT helps to reduce the variance of stochastic gradients and enhance directionality, especially when the preference gap is small, it can significantly improve training stability.

[0119] In the aforementioned training processes, to balance computational efficiency and model performance when processing high-resolution image input, a vision-language adapter is introduced. The pixel unshuffle operation divides the image into non-overlapping image tiles and performs spatial average pooling, reducing the number of visual tokens to the original value. That is, the size is The image tiles, whose visual tokens are determined by... Compress to .

[0120] In terms of image preprocessing, a dynamic resolution preprocessing strategy is adopted. Specifically, for the images used for training, the images are divided into sections based on resolution and aspect ratio. Each of the following local image tiles has a resolution of [resolution value missing]. And scale the original image to The thumbnail is used to stitch together all local image tiles and the thumbnail to form a visual input, which is then fed into a visual encoder to extract visual features.

[0121] Local image tiles are used to capture fine-grained regional information, while thumbnails are used to provide global semantic context.

[0122] For example, for high-resolution input images Divide it into sections based on resolution and aspect ratio. Each of the following local image tiles has a resolution of [resolution value missing]. ,in ,Right now At the same time, the original image is scaled down. thumbnails: The final visual input is composed of stitched together all local image tiles and thumbnails: .enter The images are fed into the InternViT visual encoder to extract visual features. These features include local image tiles. It can capture fine-grained regional information, while thumbnails It can provide global semantic context, enabling the model to take into account both local details and global patterns in high-resolution meteorological service images.

[0123] To promote the effective application of visual language models in the meteorological service field, this invention proposes a visual language model construction scheme based on high-quality meteorological service professional image-text pairs data, oriented towards meteorological service scenarios. Firstly, through domain-adaptive incremental pre-training on meteorological service data, the model acquires preliminary visual understanding capabilities for meteorological service scenarios. Subsequently, a multi-stage learning strategy combining instruction alignment and preference alignment is employed to guide the model to further enhance its instruction compliance ability and task generalization on the basis of domain-specific visual understanding capabilities, thereby improving its adaptability to various meteorological service tasks.

[0124] The visual language model training strategy proposed in this invention is optimized for typical tasks in meteorological services, which helps improve the model's adaptability and accuracy in diverse application scenarios, thereby providing a more comprehensive and precise intelligent solution for the meteorological service field. The trained visual language model can adapt to professional image understanding tasks in various meteorological service scenarios and has the ability to generate high-quality professional text descriptions based on image content under given instructions, effectively meeting the practical needs of applications such as weather forecasting and meteorological early warning services. Furthermore, this invention has good scalability, facilitating continuous model optimization during actual deployment and flexibly adapting to various segmented business scenarios.

[0125] Example 2:

[0126] This invention provides a device for constructing a multimodal large model of meteorological services, such as... Figure 4 As shown, the device includes:

[0127] Model building module 1 is used to build a visual large language model and initialize it using the pre-trained model parameters on a general visual language dataset.

[0128] The visual large language model includes a visual encoder, a visual-language adapter, and a large language model.

[0129] Incremental pre-training module 2 is used to freeze the visual encoder and the large language model. It uses the constructed meteorological service domain image-text pair dataset to perform incremental pre-training on the visual-language adapter, enabling the visual-language adapter to learn the specific semantic features and expressions of meteorological service images.

[0130] The instruction alignment module 3 is used to end the freezing of the visual encoder and the large language model. It uses a fully parameterized model training method to perform instruction alignment on the visual large language model using an instruction alignment dataset containing large-scale, diverse image-text pairs from the meteorological service domain. This enhances the visual large language model's basic ability to follow meteorological service instructions.

[0131] Preference alignment module 4 is used to freeze the visual encoder and visual-language adapter. It uses a constructed preference dataset containing rare key meteorological events to train the visual large language model with low-rank adaptive parameters, thereby enhancing the generalization ability of the visual large language model in meteorological service scenarios.

[0132] In one example, for the images used for training, the images are divided according to resolution and aspect ratio. Each of the following local image tiles has a resolution of [resolution value missing]. And scale the original image to The thumbnail is used to stitch together all local image tiles and the thumbnail to form visual input, which is then fed into a visual encoder to extract visual features.

[0133] Local image tiles are used to capture fine-grained regional information, while thumbnails are used to provide global semantic context.

[0134] When incrementally pre-training the vision-language adapter, class-balanced random sampling is used for all meteorological service domain image-text pair datasets to ensure the diversity of training data distribution.

[0135] When training the visual large language model with full parameter alignment, the instruction alignment dataset is divided into multiple ordered training stage datasets from simple to complex according to the complexity of the text data and image data. The visual large language model performs sequential learning from simple to complex according to the order of the ordered training stage datasets.

[0136] For text data, the complexity level is determined based on the length and professional complexity of the text; for image data, the complexity level is determined based on the clarity of the image background and meteorological elements; for supplementary data of new scenes and new types, they are directly assigned to the most complex ordered training stage dataset.

[0137] When training a visual large language model for preference alignment, the following loss function is used. ;

[0138]

[0139] in, To optimize loss for direct preference, To monitor and fine-tune losses, The set coefficient.

[0140] The apparatus provided in this embodiment of the invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the apparatus embodiment can be referred to the corresponding content in the aforementioned method embodiment 1. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the apparatus and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0141] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention.

Claims

1. A method for constructing a multimodal large model for meteorological services, characterized in that, The method includes: S1: Build a large visual language model and initialize it using the pre-trained model parameters on a general visual language dataset; The visual large language model includes a visual encoder, a visual-language adapter, and a large language model. S2: Freeze the visual encoder and large language model, and use the constructed meteorological service domain image-text pair dataset to incrementally pre-train the visual-language adapter, so that the visual-language adapter learns the specific semantic features and expressions of meteorological service images; S3: End the freeze on the visual encoder and the large language model, and use the constructed instruction alignment dataset containing large-scale, diverse meteorological service image-text pairs to train the visual large language model in a full-parameter model method for instruction alignment, so as to enhance the visual large language model's basic ability to follow meteorological service instructions. S4: Freeze the visual encoder and visual-language adapter, and use a preference dataset containing rare key meteorological events to train the visual large language model with low-rank adaptive parameters, thereby enhancing the generalization ability of the visual large language model in meteorological service scenarios.

2. The method for constructing a multimodal large model for meteorological services according to claim 1, characterized in that, For the images used in training, the images are divided into sections based on resolution and aspect ratio. Each of the following local image tiles has a resolution of [resolution value missing]. And scale the original image to The thumbnail is used to stitch together all local image tiles and the thumbnail into a visual input, which is then fed into a visual encoder to extract visual features. Local image tiles are used to capture fine-grained regional information, while thumbnails are used to provide global semantic context.

3. The method for constructing a multimodal large model for meteorological services according to claim 1, characterized in that, When incrementally pre-training the vision-language adapter, class-balanced random sampling is used for all meteorological service domain image-text pair datasets to ensure the diversity of training data distribution.

4. The method for constructing a multimodal large model for meteorological services according to claim 1, characterized in that, When training the visual large language model with full parameter alignment, the instruction alignment dataset is divided into multiple ordered training stage datasets from simple to complex according to the complexity of the text data and image data. The visual large language model performs sequential learning from simple to complex according to the order of the ordered training stage datasets.

5. The method for constructing a multimodal large model for meteorological services according to claim 4, characterized in that, For text data, the complexity level is determined based on the length and technical complexity of the text; for image data, the complexity level is determined based on the clarity of the image background and meteorological elements; for supplementary data of new scenes and new types, they are directly assigned to the most complex ordered training stage dataset.

6. The method for constructing a multimodal large model for meteorological services according to any one of claims 1-5, characterized in that, When training a visual large language model for preference alignment, the following loss function is used. ; in, To optimize loss for direct preference, To monitor and fine-tune losses, The set coefficient.

7. A device for constructing a multimodal large model for meteorological services, characterized in that, The device includes: The model building module is used to build a visual large language model and initialize it using the pre-trained model parameters on a general visual language dataset; The visual large language model includes a visual encoder, a visual-language adapter, and a large language model. The incremental pre-training module is used to freeze the visual encoder and the large language model. It uses the constructed meteorological service domain image-text pair dataset to perform incremental pre-training on the visual-language adapter, enabling the visual-language adapter to learn the specific semantic features and expressions of meteorological service images. The instruction alignment module is used to end the freezing of the visual encoder and the large language model. It uses a fully parameterized model training method to perform instruction alignment on the visual large language model using an instruction alignment dataset containing large-scale and diverse image-text pairs in the meteorological service domain, thereby enhancing the visual large language model's basic ability to follow meteorological service instructions. The preference alignment module is used to freeze the visual encoder and visual-language adapter. It uses a constructed preference dataset containing rare key meteorological events to train the visual large language model with low-rank adaptive parameters, thereby enhancing the generalization ability of the visual large language model in meteorological service scenarios.

Images