using unpaired data for cross-modal generation models through cycle consistency

By utilizing the ITIT framework and the cycle consistency training paradigm, and leveraging unpaired image and text data, the dependency problem of cross-modal generative models on paired datasets is solved, achieving efficient image-to-text and text-to-image generation and improving model performance.

CN122270764APending Publication Date: 2026-06-23GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
GOOGLE LLC
Filing Date
2024-09-30
Publication Date
2026-06-23

AI Technical Summary

Technical Problem

Existing cross-modal generative models rely on large paired image-text datasets during training, resulting in high data collection costs and low quality, making it difficult to effectively utilize unpaired image and text data for visual language training.

Method used

Employing the ITIT (Image-Text Integration) framework, this method utilizes a cycle consistency training paradigm to generate bidirectional images to text and text to images by combining an image-text encoder with disjoint image-text decoders, leveraging unpaired image and text data. Supervision is provided by the cycle consistency loss between the generated images/text and the original input.

Benefits of technology

It significantly reduces the reliance on paired image-text data, improves the model's performance in text-to-image and image-to-text generation, achieves similar scaling behavior to high-quality paired data, and reduces data collection costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122270764A_ABST
    Figure CN122270764A_ABST
Patent Text Reader

Abstract

A computer-implemented method includes obtaining, by a computing system that can include one or more computing devices, a text-to-image model and an image-to-text model. The method also includes, for each of one or more training iterations: accessing, by the computing system, unpaired text input; processing, by the computing system, the unpaired text input with the text-to-image model to generate a synthetic image; processing, by the computing system, the synthetic image with the image-to-text model to generate predicted text; evaluating, by the computing system, a loss function that compares the predicted text to the unpaired text input to generate a loss value; and modifying, by the computing system, one or more parameter values of one or both of the text-to-image model and the image-to-text model based on the loss function.
Need to check novelty before this filing date? Find Prior Art