A method for training multimodal large models that supports image stitching

By combining BERT and Vision Transformer models, and using hybrid multimodal fusion techniques and style loss functions to optimize image stitching, the problems of insufficient image stitching quality and semantic consistency in multimodal models are solved, achieving efficient multimodal information processing and robustness enhancement.

CN120953084BActive Publication Date: 2026-05-26CHINA NAT BUILDING MATERIALS TECH CO LTD +2
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410713205.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-04
Publication Date
2026-05-26
Estimated Expiration
2044-06-04

AI Technical Summary

Technical Problem

Existing multimodal models suffer from poor image stitching quality, poor semantic consistency, and insufficient style matching when processing image and text data. In particular, they can lead to image quality degradation and semantic inconsistency between text and images during multimodal information fusion.

Method used

We employ a hybrid multimodal fusion technique, combining the BERT architecture and the Vision Transformer visual model. We use a style loss function and a discriminator to evaluate the quality of the generated images, implement end-to-end training, and optimize the model through a learning rate scheduler and data augmentation techniques. This includes multimodal feature alignment, CycleGAN architecture, cross-modal attention mechanism, multi-head attention mechanism, fully connected layers, and deconvolutional layers to design the generative network.

Benefits of technology

It achieves efficient image stitching and multimodal information processing, improves the accuracy and consistency of image stitching, ensures the consistency of visual style and richness of detail in the generated images, and enhances the robustness and generalization ability of the model when processing complex multimodal data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120953084B_ABST
    Figure CN120953084B_ABST
Patent Text Reader

Abstract

This invention relates to the field of multimodal model training technology, specifically, to a method for training a large multimodal model supporting image stitching. It includes the following steps: S1, using hybrid multimodal fusion technology, combining the BERT architecture and the Vision Transformer visual model to construct an image stitching multimodal model; S2, using a style loss function to adjust the style of the generated image to achieve high-quality style transfer; S3, using a primary discriminator and an auxiliary discriminator to evaluate the quality of the generated image, calculating the discriminator loss and auxiliary discriminator loss to optimize the stitched image; S4, implementing end-to-end training, using a learning rate scheduler and data augmentation techniques to improve model efficiency. This method for training a large multimodal model supporting image stitching achieves efficient fusion of text and image data through hybrid multimodal fusion technology, adjusts the style of the generated image through a style loss function, and improves the accuracy and visual consistency of image stitching through end-to-end training.
Need to check novelty before this filing date? Find Prior Art

Citation Information

Patent Citations

  • Cross-domain image style migration method based on semantic GAN

    CN114359526A

  • Fashion commodity image generation method based on attention generative adversarial network

    CN117636106A