Fine adjustment method and device for text graph model, electronic equipment and storage medium

Through the two-stage training method, the keyword weight of the text encoder and the freezing text encoder are adjusted, which solves the problems of insufficient learning ability and overfitting in fine-tuning of the literary graphics model, and improves the efficiency and quality of image generation.

CN119312842AActive Publication Date: 2025-01-14BEIJING SHENGSHU TECH CO LTD
View PDF 5 Cites 2 Cited by

Patent Information

Application Number
CN202411382118.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-30
Publication Date
2025-01-14
Estimated Expiration
2044-09-30

AI Technical Summary

Technical Problem

The traditional fine-tuning method of literary and biographical graphics models only fine-tune the image diffusion model, freezing the text encoder, limiting the model's learning ability, resulting in overfitting and inefficiency in the generated images.

Method used

The two-stage training method is adopted. First, the text encoder and diffusion generation model are trained at the same time, the keyword weight of the text encoder is adjusted, and the training efficiency is improved; secondly, the text encoder is frozen, and the diffusion generation model is adjusted only to avoid overfitting the text encoder. Ensure that the model generates high-quality images.

Benefits of technology

It improves the learning ability and generation efficiency of the image generation model, reduces overfitting behavior, avoids "burning effect" and "catastrophic forgetting", and improves the robustness and generation quality of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119312842A_ABST
    Figure CN119312842A_ABST
Patent Text Reader

Abstract

The invention relates to a fine adjustment method and device of a text graph model, electronic equipment and a storage medium. The method comprises the following steps: acquiring a preset image training set, wherein the preset image training set comprises a first image and a first text corresponding to the first image; the method comprises the following steps: training a preset text graph model according to a preset image training set, training the text encoder and the diffusion generation model at the same time in a first training stage, freezing the text encoder trained in the first stage in a second training stage, and training the diffusion generation model, a target text graph model is obtained, and the target text graph model is used for outputting a first image according to the first text; and obtaining a user image text according to a user input instruction, and obtaining a target image based on the user image text and the target text generation graph model. According to the scheme provided by the invention, the learning ability of the image generation model is enhanced, and the over-fitting behavior of training the image generation model in the image generation process is reduced.
Need to check novelty before this filing date? Find Prior Art

Citation Information

Patent Citations

  • Model training method and device, image generation method and electronic equipment

    CN117333880A

  • General feature map acquisition method and related equipment

    CN117649578A

  • Method and system for generating facet defect image sample

    CN118247604A

  • Figure generation method, model training method and device, electronic equipment and storage medium

    CN118447511A

  • Training and deployment of image generation models

    US11995803B1

Cited By

  • Unmanned aerial vehicle performance pattern generation method and device, computer equipment and storage medium

    CN120747288A

  • Unmanned aerial vehicle performance pattern generation method and device, computer device, and storage medium

    CN120747288B