Multi-modal large model training method supporting picture splicing

Abstract

The invention relates to the technical field of multi-modal model training, in particular to a multi-modal large model training method supporting picture splicing. The method comprises the following steps: S1, constructing an image splicing multi-modal model by using a hybrid multi-modal fusion technology in combination with a BERT framework and a Vision Transform visual model; s2, adjusting the style of the generated image by using a style loss function to realize high-quality style migration; s3, evaluating the quality of the generated image by using a main discriminator and an auxiliary discriminator, and calculating the loss of the discriminator and the loss of the auxiliary discriminator to optimize the spliced image; and S4, implementing end-to-end training, and improving the model efficiency by using a learning rate scheduler and a data enhancement technology. According to the multi-modal large model training method supporting picture splicing, efficient fusion of text and image data is realized by using a hybrid multi-modal fusion technology, and the style of the generated image and end-to-end training are adjusted through a style loss function, so that the accuracy and visual consistency of image splicing are improved.

Description

Technical Field

[0001] This invention relates to the field of multimodal model training technology, and more specifically, to a method for training large multimodal models that supports image stitching. Background Technology

[0002] A multimodal large model training method that supports image stitching is based on multimodal fusion technology, which is used to automatically extract and stitch images from text and image data. It aims to achieve efficient multimodal information fusion and processing through BERT language model and Vision Transformer visual model.

[0003] Existing multimodal models suffer from poor image stitching quality, poor semantic consistency, and insufficient style matching when processing image and text data. Due to the complexity and diversity of image and text data, image quality deterioration and semantic inconsistency between text and images can occur during multimodal information fusion. Therefore, this paper proposes a method for training large multimodal models that supports image stitching. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal large model training method that supports image stitching, in order to solve the problems mentioned in the background art, which are caused by the complexity and diversity of image and text data, leading to image quality degradation and semantic inconsistency between text and images during multimodal information fusion.

[0005] To achieve the above objectives, the present invention aims to provide a method for training a multimodal large model that supports image stitching, comprising the following steps:

[0006] S1. Using hybrid multimodal fusion technology, combined with the BERT architecture and the Vision Transformer visual model, an image stitching multimodal model is constructed.

[0007] S2. Use a style loss function to adjust the style of the generated image to achieve style transfer;

[0008] S3. Use the primary discriminator and the secondary discriminator to evaluate the quality of the generated image and calculate the discriminator loss L. D And auxiliary discriminator loss L G Optimize the stitched image and process the generated stitched image I. gen Perform noise reduction and enhancement;

[0009] S4. Implement end-to-end training, using a learning rate scheduler and data augmentation techniques to improve model efficiency.

[0010] As a further improvement to this technical solution, the hybrid multimodal fusion technology is a technology that integrates multiple advanced methods for efficiently fusing text and images from different modalities, including multimodal feature alignment technology, CycleGAN architecture, image block embedding, cross-modal attention mechanism, multi-head attention mechanism, fully connected layers and activation functions, classifier and deconvolution layer design to generate networks;

[0011] The image stitching multimodal model is based on hybrid multimodal fusion technology and is used to integrate text and image data to achieve efficient image stitching and multimodal information processing functions. The image stitching multimodal model includes a hybrid input module, an image stitching module, a multimodal fusion layer, and an output layer.

[0012] As a further improvement to this technical solution, the steps for constructing the image stitching multimodal model in S1 are as follows:

[0013] S1.1 Hybrid Input Module: Processes the input text X and image separately, extracts features from the processed text and image, and uses multimodal feature alignment technology to unify the dimensions and representation of text feature T and image feature I to obtain a fused feature vector F, which is then output to the next layer;

[0014] S1.2, Image stitching module: Decodes the fused feature vector F into new image features G. F Using the CycleGAN architecture, a generator G processes and stitches together images, generating a stitched image I. gen The quality of the generated image is evaluated and optimized using discriminator D, and the generated stitched image I is then processed. gen Perform noise reduction and enhancement;

[0015] S1.3, Multimodal Fusion Layer: From the stitched image I gen The stitched image features G extracted from gen Using a cross-modal attention mechanism, text features T are used as the query, and the decoded new image features G are... F and spliced ​​image features G gen As keys and values, intermediate fusion feature A is generated. fusion A multi-head attention mechanism is used to concatenate the outputs of multiple attention heads and generate the final fused feature F through a linear transformation. final ;

[0016] S1.4, Output Layer: Based on the final fusion feature F final To perform classification and generation tasks, a classifier is designed using fully connected layers and activation functions to obtain the classification result C. output Image G is generated using a deconvolutional layer design generative network. output .

[0017] As a further improvement to this technical solution, the specific steps involved in the mixed input module in S1.1 to process the input text and image separately and output them to the next layer are as follows:

[0018] S1.1.1 Initialize and load the BERT architecture weights, segment the input text X into lexical units C using the Byte-Pair Encoding algorithm, and then use word embedding technology to encode the segmentation result into a vector E. C :

[0019] C = BPE(X);

[0020] E C =E[C];

[0021] Where X is the input text; BPE is the Byte-Pair Encoding segmentation operation; C is the segmented vocabulary unit; E is the word embedding matrix; E C It is an encoded vector;

[0022] S1.1.2 Initialize and load the Vision Transformer visual model weights, and perform scaling, normalization, and Patch Embedding preprocessing operations on the input image;

[0023] S1.1.3, the encoded vector E C The text features T are input into the BERT model, and the preprocessed image is input into the Vision Transformer visual model to obtain the image features I.

[0024] S1.1.4. Using multimodal feature alignment technology, the dimensions of text feature T and image feature I are unified to obtain unified text feature T′ and unified image feature I′, which are then concatenated into a fused feature vector F:

[0025]

[0026] in, N is the sequence length of the text features; M is the number of patches for the image features; d′ is the dimension of the aligned features; and the dimension of F is (N+M)×d′.

[0027] S1.1.5. Pass the obtained fused feature vector F to the multimodal fusion layer.

[0028] Among them, the Byte-Pair Encoding algorithm in S1.1.1 is a frequency-based sub-word segmentation algorithm that aims to reduce the size of the vocabulary while maintaining the representability of the text. It generates sub-word units by repeatedly merging the most frequent character pairs.

[0029] In S1.1.1, word embedding technology is a technique that maps words to a high-dimensional vector space. Commonly used methods include Word2Vec, GloVe, and Transformer-based embedding, which converts sub-word units into vector representations through a pre-trained word embedding matrix.

[0030] In S1.1.2, image scaling adjusts the input image to a standard size, and normalization adjusts the image pixel values ​​to a uniform range (usually 0 to 1) to adapt to the input requirements of the model. Commonly used normalization methods include mean normalization and standardization.

[0031] In S1.1.2, Patch Embedding divides the image into fixed-size image patches. Each patch is flattened and linearly transformed into an embedding vector, which serves as the input to the ViT model.

[0032] As a further improvement to this technical solution, the image stitching module in S1.2 uses a stitched image I generated by the CycleGAN architecture. gen The specific steps involved are as follows:

[0033] S1.2.1 Decode the fused feature vector F into new image features G. F :

[0034] G F =Decoder(F);

[0035] Here, Decoder is a decoder function used to convert feature vectors into image features;

[0036] S1.2.2 Initialize the CycleGAN generator G by decoding the image features G F The stitched image I generated as input to the generator gen :

[0037] I gen =G(G F );

[0038] Where G is the generator of CycleGAN.

[0039] As a further improvement to this technical solution, in S1.3, the multimodal fusion layer uses a cross-modal attention mechanism to generate intermediate fusion features A. fusion The mathematical formulas involved are as follows:

[0040]

[0041] in, It is the image feature after decoding the fused feature vector F; G gen It is a stitched image I gen Image features;

[0042] The multimodal fusion layer uses a multi-head attention mechanism and generates the final fused feature F through linear transformation. final The mathematical formulas involved are as follows:

[0043] head i =Attention(T′W i Q G F W i K G gen W i V );

[0044] F final =Concat(head1, head2,..., head h W o ;

[0045] Among them, T′W i Q For querying matrix Q; W i Q To query the weight matrix; G F W i K Let K be the bond matrix; W i K G is the key weight matrix; gen W i V Value matrix V; W i V Value weight matrix; head i For the output of a single attention head; Concat is a concatenation operation that concatenates the outputs of all attention heads line by line; W o It is the output weight matrix.

[0046] As a further improvement to this technical solution, in S1.4, the output layer determines the final fusion feature F. final To perform classification and generation tasks, a classifier is designed using fully connected layers and activation functions to obtain the classification result C. output The mathematical formulas involved are as follows:

[0047] C output =softmax(W c F final +b c );

[0048] Wherein, the Wc b is the weight matrix of the classifier; c C is the bias vector of the classifier; output This is the classification result;

[0049] The output layer uses a deconvolutional layer to design a generative network that generates image G. output The mathematical formulas involved are as follows:

[0050] G output =Generator(F final );

[0051] Where Generator is the generator network; G output It generates an image.

[0052] As a further improvement to this technical solution, the style loss function in S2 is used for the style transfer task. By calculating the difference between the generated image and the target style image, the output of the generator is adjusted to make the generated image style-close to the target image. The specific steps involved in using the style loss function to adjust the style of the generated image are as follows:

[0053] S2.1, For the generated image G output Calculate the Gram matrix from the feature map of layer l:

[0054]

[0055] S2.2, For the target style image G style Calculate the Gram matrix from the feature map of layer l:

[0056]

[0057] S2.3 Calculate the style loss between the generated image and the target style image at layer l:

[0058]

[0059] S2.4 Calculate the total style loss:

[0060] L style =∑ l w l ·L style，l ;

[0061] Among them, G output To generate an image; G style For the target style image; φ l G is the feature extraction function for the l-th layer; G(·) is the Gram matrix calculation function; The Frobenius norm is used to measure the difference between two matrices; l is the number of network layers; N is the number of channels; M is the spatial dimension of the feature map; w l represents the weight of the l-th layer.

[0062] As a further improvement to this technical solution, in step S3, the main discriminator is used to improve the discriminator's ability to distinguish between real and generated images; the auxiliary discriminator is used to evaluate the semantic and style fit of the generated image and calculate the loss L of the main discriminator. D And auxiliary discriminator loss L G The optimization involves the following mathematical model formulas.

[0063] Master discriminator loss L D :

[0064]

[0065] Auxiliary discriminator loss L G :

[0066]

[0067] Total loss L:

[0068] L = L D +L G ;

[0069] Among them, I real The image is a real image; D is the primary discriminator; G is the auxiliary discriminator; and L is the total loss.

[0070] The denoising and enhancement of the generated stitched image are used to reduce noise in the generated image, improve image quality, and enhance visual quality and detail. The denoising and enhancement operations include filtering, denoising networks, contrast adjustment, brightness adjustment, and sharpening filtering. Through these steps, the generated stitched image is not only optimized in terms of realism and semantic consistency, but also further improved in visual quality and detail through denoising and enhancement techniques.

[0071] As a further improvement to this technical solution, in step S4, the learning rate scheduler is used to monitor the verification loss and automatically adjust the learning rate. The mathematical model formula involved is as follows:

[0072] η new =η×factor if no improvement in patience epochs;

[0073] Where η is the current learning rate; factor is the decay factor;

[0074] The data augmentation technique is used to improve the generalization ability and robustness of the image stitching multimodal model. The data augmentation technique operations include random cropping and rotation, and the mathematical model formulas involved in random cropping and rotation are as follows:

[0075] Random cropping: I aug =RandomCrop(I gen );

[0076] Rotation: I aug =RandomRotation(I gen );

[0077] Among them, I aug For the enhanced image; RandomCrop(I gen ) is the function for random cropping; RandomRotation(I gen ) is the random rotation operation function.

[0078] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0079] 1. In this multimodal large model training method that supports image stitching, the use of hybrid multimodal fusion technology can efficiently fuse text and image data, thereby enhancing the accuracy and consistency of image stitching.

[0080] 2. In this multimodal large model training method that supports image stitching, by using a style loss function to adjust the style of the generated images and implementing comprehensive end-to-end training, the consistency of the generated images in visual style and the richness of details can be ensured, while improving the robustness and generalization ability of the model when processing complex multimodal data. Attached Figure Description

[0081] Figure 1 This is a flowchart of the overall method of the present invention. Detailed Implementation

[0082] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0083] Example:

[0084] Please see Figure 1 As shown, this embodiment provides a method for training a multimodal large model that supports image stitching, including the following steps:

[0085] S1. Using hybrid multimodal fusion technology, combined with the BERT architecture and the Vision Transformer visual model, an image stitching multimodal model is constructed.

[0086] The hybrid multimodal fusion technology is a technique that integrates multiple advanced methods for efficiently fusing text and images from different modalities. These methods include multimodal feature alignment, Cyc|eGAN architecture, image block embedding, cross-modal attention mechanism, multi-head attention mechanism, fully connected layers and activation functions, classifiers and deconvolutional layers to design and generate networks.

[0087] The image stitching multimodal model is based on hybrid multimodal fusion technology and is used to integrate text and image data to achieve efficient image stitching and multimodal information processing functions. The image stitching multimodal model includes a hybrid input module, an image stitching module, a multimodal fusion layer, and an output layer.

[0088] The steps for constructing the image stitching multimodal model in S1 are as follows:

[0089] S1.1 Hybrid Input Module: Processes the input text X and image separately, extracts features from the processed text and image, and uses multimodal feature alignment technology to unify the dimensions and representation of text feature T and image feature I to obtain a fused feature vector F, which is then output to the next layer;

[0090] S1.2, Image stitching module: Decodes the fused feature vector F into new image features G. F Using the CycleGAN architecture, a generator G processes and stitches together images, generating a stitched image I. gen The quality of the generated image is evaluated and optimized using discriminator D, and the generated stitched image I is then processed. gen Perform noise reduction and enhancement;

[0091] S1.3, Multimodal Fusion Layer: From the stitched image I gen The stitched image features G extracted from gen Using a cross-modal attention mechanism, text features T are used as the query, and the decoded new image features G are... F and spliced ​​image features G gen As keys and values, intermediate fusion feature A is generated. fusion A multi-head attention mechanism is used to concatenate the outputs of multiple attention heads and generate the final fused feature F through a linear transformation. final ;

[0092] S1.4, Output Layer: Based on the final fusion feature F final To perform classification and generation tasks, a classifier is designed using fully connected layers and activation functions to obtain the classification result C. outputImage G is generated using a deconvolutional layer design generative network. output ;

[0093] The specific steps involved in the mixed input module in S1.1 to process the input text and image separately and output them to the next layer are as follows:

[0094] S1.1.1 Initialize and load the BERT architecture weights, segment the input text X into lexical units C using the Byte-Pair Encoding algorithm, and then use word embedding technology to encode the segmentation result into a vector E. C :

[0095] C = BPE(X);

[0096] E C =E[C];

[0097] Where X is the input text; BPE is the Byte-Pair Encoding segmentation operation; C is the segmented vocabulary unit; E is the word embedding matrix; E C It is an encoded vector;

[0098] S1.1.2 Initialize and load the Vision Transformer visual model weights, and perform scaling, normalization, and Patch Embedding preprocessing operations on the input image;

[0099] S1.1.3, the encoded vector E C The text features T are input into the BERT model, and the preprocessed image is input into the Vision Transformer visual model to obtain the image features I.

[0100] S1.1.4. Using multimodal feature alignment technology, the dimensions of text feature T and image feature I are unified to obtain unified text feature T' and unified image feature I', which are then concatenated into a fused feature vector F:

[0101]

[0102] in, N is the sequence length of the text features; M is the number of patches for the image features; d′ is the dimension of the aligned features; and the dimension of F is (N+M)×d′.

[0103] S1.1.5. Pass the obtained fused feature vector F to the multimodal fusion layer;

[0104] Among them, the Byte-Pair Encoding algorithm in S1.1.1 is a frequency-based sub-word segmentation algorithm that aims to reduce the size of the vocabulary while maintaining the representability of the text. It generates sub-word units by repeatedly merging the most frequent character pairs.

[0105] In S1.1.1, word embedding technology is a technique that maps words to a high-dimensional vector space. Commonly used methods include Word2Vec, GloVe, and Transformer-based embedding, which converts sub-word units into vector representations through a pre-trained word embedding matrix.

[0106] In S1.1.2, image scaling adjusts the input image to a standard size, and normalization adjusts the image pixel values ​​to a uniform range to adapt to the input requirements of the model. Commonly used normalization methods include mean normalization and standardization.

[0107] In S1.1.2, Patch Embedding divides the image into fixed-size image patches. Each patch is flattened and linearly transformed into an embedding vector, which serves as the input to the ViT model.

[0108] The image stitching module in S1.2 uses the CycleGAN architecture to generate the stitched image I. gen The specific steps involved are as follows:

[0109] S1.2.1 Decode the fused feature vector F into new image features G. F :

[0110] G F =Decoder(F);

[0111] Here, Decoder is a decoder function used to convert feature vectors into image features;

[0112] S1.2.2 Initialize the CycleGAN generator G by decoding the image features G F The stitched image I generated as input to the generator gen :

[0113] I gen =G(G F );

[0114] Where G is the generator of CycleGAN.

[0115] In step S1.3, the multimodal fusion layer uses a cross-modal attention mechanism to generate intermediate fusion features A. fusion The mathematical formulas involved are as follows:

[0116]

[0117] in, It is the image feature after decoding the fused feature vector F; G gen It is a stitched image I gen Image features;

[0118] The multimodal fusion layer uses a multi-head attention mechanism and generates the final fused feature F through linear transformation. final The mathematical formulas involved are as follows:

[0119] head i =Attention(T′W i Q G F W i K G gen W i V );

[0120] F final =Concat(head1, head2,..., head h W o ;

[0121] Among them, T′W i Q For querying matrix Q; W i Q To query the weight matrix; G F W i K Let K be the bond matrix; W i K G is the key weight matrix; gen W i V Value matrix V; W i V Value weight matrix; head i For the output of a single attention head; Concat is a concatenation operation that concatenates the outputs of all attention heads line by line; W o It is the output weight matrix.

[0122] In S1.4, the output layer is based on the final fusion feature F final To perform classification and generation tasks, a classifier is designed using fully connected layers and activation functions to obtain the classification result C. output The mathematical formulas involved are as follows:

[0123] C output =softmax(W c F final +bc );

[0124] Wherein, the W c b is the weight matrix of the classifier; c C is the bias vector of the classifier; output This is the classification result;

[0125] The output layer uses a deconvolutional layer to design a generative network that generates image G. output The mathematical formulas involved are as follows:

[0126] G output =Generator(F final );

[0127] Where Generator is the generator network; G output It generates an image.

[0128] S2. Use a style loss function to adjust the style of the generated image to achieve style transfer;

[0129] The style loss function in S2 is used for the style transfer task. It adjusts the generator's output by calculating the difference between the generated image and the target style image, so that the generated image is stylistically close to the target image. The specific steps involved in adjusting the style of the generated image using the style loss function are as follows:

[0130] S2.1, For the generated image G output Calculate the Gram matrix from the feature map of layer l:

[0131]

[0132] S2.2, For the target style image G style Calculate the Gram matrix from the feature map of layer l:

[0133]

[0134] S2.3 Calculate the style loss between the generated image and the target style image at layer l:

[0135]

[0136] S2.4 Calculate the total style loss:

[0137] L style =∑ l w l ·L style，l ;

[0138] Among them, G output To generate an image; G styleThe target style image is φl; the l-th layer feature extraction function is G(·); and the Gram matrix calculation function is G(·). The Frobenius norm is used to measure the difference between two matrices; l is the number of network layers; N is the number of channels; M is the spatial dimension of the feature map; w l represents the weight of the l-th layer.

[0139] S3. Use the primary discriminator and the secondary discriminator to evaluate the quality of the generated image and calculate the discriminator loss L. D And auxiliary discriminator loss L G Optimize the stitched image and process the generated stitched image I. gen Perform noise reduction and enhancement;

[0140] In step S3, the main discriminator is used to improve the discriminator's ability to distinguish between real and generated images; the auxiliary discriminator is used to evaluate the semantic and style fit of the generated image and calculate the loss L of the main discriminator. D And auxiliary discriminator loss L G The optimization involves the following mathematical model formulas.

[0141] Master discriminator loss L D :

[0142]

[0143] Auxiliary discriminator loss L G :

[0144]

[0145] Total loss L:

[0146] L = L D +L G ;

[0147] Among them, I real The image is a real image; D is the primary discriminator; G is the auxiliary discriminator; and L is the total loss.

[0148] The denoising and enhancement of the generated stitched image are used to reduce noise in the generated image, improve image quality, and enhance visual quality and detail. The denoising and enhancement operations include filtering, denoising networks, contrast adjustment, brightness adjustment, and sharpening filtering. Through these steps, the generated stitched image is not only optimized in terms of realism and semantic consistency, but also further improved in visual quality and detail through denoising and enhancement techniques.

[0149] S4. Implement end-to-end training, using a learning rate scheduler and data augmentation techniques to improve model efficiency;

[0150] In step S4, the learning rate scheduler is used to monitor the validation loss and automatically adjust the learning rate. The mathematical model formula involved is as follows:

[0151] η new =η×factor if no improvement in patience epochs;

[0152] Where η is the current learning rate; factor is the decay factor;

[0153] The data augmentation technique is used to improve the generalization ability and robustness of the image stitching multimodal model. The data augmentation technique operations include random cropping and rotation, and the mathematical model formulas involved in random cropping and rotation are as follows:

[0154] Random cropping: I aug =RandomCrop(I gen );

[0155] Rotation: I aug =RandomRotation(I gen );

[0156] Among them, I aug For the enhanced image; RandomCrop(I gen ) is the function for random cropping; RandomRotation(I gen ) is the random rotation operation function.

[0157] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for training a multimodal large model that supports image stitching, characterized in that: Includes the following steps: S1. Using hybrid multimodal fusion technology, combined with the BERT architecture and the Vision Transformer visual model, an image stitching multimodal model is constructed; S2. Use a style loss function to adjust the style of the generated image to achieve style transfer; S3. Use the primary discriminator and the secondary discriminator to evaluate the quality of the generated image and calculate the discriminator loss L. D And auxiliary discriminator loss L G Optimize the stitched image and process the generated stitched image I. gen Perform noise reduction and enhancement; S4. Implement end-to-end training, using a learning rate scheduler and data augmentation techniques to improve model efficiency.

2. The multimodal large model training method supporting image stitching according to claim 1, characterized in that: The hybrid multimodal fusion technology is a technique that integrates multiple advanced methods for efficiently fusing text and images from different modalities. These methods include multimodal feature alignment, CycleGAN architecture, image block embedding, cross-modal attention mechanism, multi-head attention mechanism, fully connected layers and activation functions, classifiers and deconvolutional layers to design generative networks. The image stitching multimodal model is based on hybrid multimodal fusion technology and is used to integrate text and image data to achieve efficient image stitching and multimodal information processing functions. The image stitching multimodal model includes a hybrid input module, an image stitching module, a multimodal fusion layer, and an output layer.

3. The multimodal large model training method supporting image stitching according to claim 2, characterized in that: The steps for constructing the image stitching multimodal model in S1 are as follows: S1.1 Hybrid Input Module: Processes the input text X and image separately, extracts features from the processed text and image, and uses multimodal feature alignment technology to unify the dimensions and representation of text feature T and image feature I to obtain a fused feature vector F, which is then output to the next layer; S1.2, Image stitching module: Decodes the fused feature vector F into new image features G. F Using the CycleGAN architecture, a generator G processes and stitches together images, generating a stitched image I. gen The quality of the generated image is evaluated and optimized using discriminator D, and the generated stitched image I is then processed. gen Perform noise reduction and enhancement; S1.3, Multimodal Fusion Layer: From the stitched image I gen The stitched image features G extracted from gen Using a cross-modal attention mechanism, text features T are used as the query, and the decoded new image features G are... F and spliced ​​image features G gen As keys and values, intermediate fusion feature A is generated. fusion A multi-head attention mechanism is used to concatenate the outputs of multiple attention heads and generate the final fused feature F through a linear transformation. final ; S1.4, Output Layer: Based on the final fusion feature F final To perform classification and generation tasks, a classifier is designed using fully connected layers and activation functions to obtain the classification result C. output Image G is generated using a deconvolutional layer design generative network. output .

4. The multimodal large model training method supporting image stitching according to claim 3, characterized in that: The specific steps involved in the mixed input module in S1.1 to process the input text and image separately and output them to the next layer are as follows: S1.1.1 Initialize and load the BERT architecture weights, segment the input text X into lexical units C using the Byte-Pair Encoding algorithm, and then use word embedding technology to encode the segmentation result into a vector E. C : C = BPE(X); AND C =E[C]; Where X is the input text; BPE is the Byte-Pair Encoding segmentation operation; C is the segmented vocabulary unit; E is the word embedding matrix; E C It is an encoded vector; S1.1.2 Initialize and load the Vision Transformer visual model weights, and perform scaling, normalization, and Patch Embedding preprocessing operations on the input image; S1.1.3, the encoded vector E C The text features T are input into the BERT model, and the preprocessed image is input into the Vision Transformer visual model to obtain the image features I. S1.1.

4. Using multimodal feature alignment technology, the dimensions of text feature T and image feature I are unified to obtain unified text feature T′ and unified image feature I′, which are then concatenated into a fused feature vector F: in, N is the sequence length of the text features; M is the number of patches for the image features; d′ is the dimension of the aligned features; and the dimension of F is (N+M)×d′. S1.1.

5. Pass the obtained fused feature vector F to the multimodal fusion layer.

5. The multimodal large model training method supporting image stitching according to claim 3, characterized in that: The image stitching module in S1.2 uses the CycleGAN architecture to generate the stitched image I. gen The specific steps involved are as follows: S1.2.1 Decode the fused feature vector F into new image features G. F : G F =Decoder(F); Here, Decoder is a decoder function used to convert feature vectors into image features; S1.2.2 Initialize the CycleGAN generator G by decoding the image features G F The stitched image I generated as input to the generator gen : I gen =G(G F ); Where G is the generator of CycleGAN.

6. The multimodal large model training method supporting image stitching according to claim 3, characterized in that: In step S1.3, the multimodal fusion layer uses a cross-modal attention mechanism to generate intermediate fusion features A. fusion The mathematical formulas involved are as follows: in, It is the image feature after decoding the fused feature vector F; G gen It is a stitched image I gen Image features; The multimodal fusion layer uses a multi-head attention mechanism and generates the final fused feature F through linear transformation. final The mathematical formulas involved are as follows: F final =Concat(head1,head2,...,head h )W o ; Among them, T′W i Q For querying matrix Q; W i Q To query the weight matrix; G F W i K Let K be the bond matrix; W be the bond matrix. i K G is the key weight matrix; gen W i V Value matrix V; W i V Value weight matrix; head i For the output of a single attention head; Concat is a concatenation operation that concatenates the outputs of all attention heads line by line; W o It is the output weight matrix.

7. The multimodal large model training method supporting image stitching according to claim 3, characterized in that: In S1.4, the output layer is based on the final fusion feature F final To perform classification and generation tasks, a classifier is designed using fully connected layers and activation functions to obtain the classification result C. output The mathematical formulas involved are as follows: C output =softmax(W c F final +b c ); Wherein, the W c b is the weight matrix of the classifier; c C is the bias vector of the classifier; output This is the classification result; The output layer uses a deconvolutional layer to design a generative network that generates image G. output The mathematical formulas involved are as follows: G output =Generator(F final ); Where Generator is the generator network; G output It generates an image.

8. The multimodal large model training method supporting image stitching according to claim 1, characterized in that: The style loss function in S2 is used for the style transfer task. It adjusts the generator's output by calculating the difference between the generated image and the target style image, so that the generated image is stylistically close to the target image. The specific steps involved in adjusting the style of the generated image using the style loss function are as follows: S2.1, For the generated image G output Calculate the Gram matrix from the feature map of layer l: S2.2, For the target style image G style Calculate the Gram matrix from the feature map of layer l: S2.3 Calculate the style loss between the generated image and the target style image at layer l: S2.4 Calculate the total style loss: L style =∑ l w l ·L style，l ; Among them, G output To generate an image; G style For the target style image; φ l G is the feature extraction function for the l-th layer; G(·) is the Gram matrix calculation function; The Frobenius norm is used to measure the difference between two matrices; l is the number of network layers; N is the number of channels; M is the spatial dimension of the feature map; w l represents the weight of the l-th layer.

9. The multimodal large model training method supporting image stitching according to claim 1, characterized in that: In step S3, the main discriminator is used to improve the discriminator's ability to distinguish between real images and generated images; The auxiliary discriminator is used to evaluate the semantic and style fit of the generated image, and calculates the loss L of the main discriminator. D And auxiliary discriminator loss L G The optimization involves the following mathematical model formulas. Master discriminator loss L D : Auxiliary discriminator loss L G : Total loss L: L=L D +L G ; Among them, I real It is a real image; D is the primary discriminator; G is the auxiliary discriminator; and L is the total loss.

10. The multimodal large model training method supporting image stitching according to claim 1, characterized in that: In step S4, the learning rate scheduler is used to monitor the validation loss and automatically adjust the learning rate. The mathematical model formula involved is as follows: η new =η×factor if no improvement in patience epochs; Where η is the current learning rate; factor is the decay factor; The data augmentation technique is used to improve the generalization ability and robustness of the image stitching multimodal model. The data augmentation technique operations include random cropping and rotation, and the mathematical model formulas involved in random cropping and rotation are as follows: Random cropping: I aug =RandomCrop(I gen ); Rotation: I aug =RandomRotation(I gen ); Among them, I aug For the enhanced image; RandomCrop(I gen ) is the function for random cropping; RandomRotation(I gen ) is the random rotation operation function.

Images