Low-cost high-precision image modeling and reconstruction method
By employing a low-cost, high-precision image modeling and reconstruction method, and utilizing a visual encoder and BLIP decoder to enhance the tokenizer representation capability, this approach solves the problems of high training costs and detail loss in image generation methods, achieving highly efficient image reconstruction results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHONGBEI UNIV
- Filing Date
- 2026-01-16
- Publication Date
- 2026-05-01
AI Technical Summary
Existing image generation methods suffer from high training costs and large computational demands, and the discrete visual tokenizer leads to the loss of image detail information, making it difficult to meet the needs of practical applications.
We employ a low-cost, high-precision image modeling and reconstruction method. We extract and quantize feature maps through a visual encoder, combine a BLIP decoder and a discriminator DINO v2, introduce an auxiliary task to enhance the representational capability of the tokenizer, and use text modalities to supplement image information to form a branch supervision signal.
While reducing training costs and data requirements, it significantly improves image detail preservation and reconstruction quality, thereby increasing model efficiency and practical value.
Smart Images
Figure CN121962344A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, specifically relating to a low-cost, high-precision image modeling and reconstruction method. Background Technology
[0002] In recent years, with the rapid development of multimodal large models, the functions they support have been increasing and gradually becoming more widespread, among which image generation technology has attracted particular attention. Most existing methods rely primarily on diffusion models. While these methods have achieved some success, they still suffer from unavoidable drawbacks: the need to model continuous image variables in the latent space leads to excessively high model training costs. Existing image generation methods mainly rely on diffusion models. Although diffusion models have achieved some success in the field of image generation, their training costs are extremely high, computational demands are enormous, and overall efficiency is low, making large-scale deployment in commercial applications difficult.
[0003] In contrast, large language models typically process data by modeling it as a series of discrete tokens and predicting the next token sequentially. However, existing discrete visual tokenizers, due to design limitations, inevitably suffer from the loss of image detail. This information loss directly affects the stability and accuracy of downstream tasks, ultimately leading to misalignment and distortion in the generated or reconstructed images, making it difficult to meet practical application requirements.
[0004] To address these issues, researchers have proposed a promising development path: representing images as a finite and discrete set of tokens using a tokenizer, followed by large-scale language models for subsequent modeling and processing. However, existing tokenizers have limited representational capabilities, often losing detailed information in the image (such as facial features and text content), resulting in poor performance of downstream large-scale models during the modeling and generation stages. Furthermore, existing methods face significant bottlenecks in terms of data requirements: most tokenizers require training on at least five million, or even billions, of images, and collecting such large datasets legally and compliantly is extremely difficult. Therefore, how to train a tokenizer with rich and detailed representation capabilities at a lower cost has become an urgent problem to be solved.
[0005] Current methods mostly focus on modeling the semantics of image pixels. However, during discretization, the semantics of consecutive pixels are inevitably lost, leading to a weakening of image detail. Numerous experiments have shown that the redundancy of image information is far greater than that of text information. Summary of the Invention
[0006] To address the problem that the loss of semantics of consecutive pixels during the discretization process in existing image processing techniques leads to a weakening of image detail information, this invention provides a low-cost, high-precision image modeling and reconstruction method.
[0007] To achieve the above objectives, the present invention employs the following technical solutions:
[0008] A low-cost, high-precision image modeling and reconstruction method, the method comprising the following steps:
[0009] Step 1: Collect an image dataset, filter and input images and their corresponding text descriptions from the dataset, and preprocess the input images to a uniform size;
[0010] Step 2: Construct a low-cost, high-precision image modeling and reconstruction model, use the visual encoder in the model to extract features from the preprocessed image to obtain visual feature maps in continuous space, and then fuse the visual feature maps;
[0011] The specific operation of step 2 is as follows:
[0012] Step 2.1: Process the visual feature map through the attention projection layer in the model, fuse global context information, and obtain a continuous spatial feature map;
[0013] Step 2.2: Compress the continuous spatial feature map obtained in Step 2.1 using a vector compression method to obtain a discrete vector with the closest cosine similarity to the continuous vector, thereby realizing the quantization of the visual feature map;
[0014] Step 2.3: Perform overall information fusion on the quantized visual feature map through another attention projection layer in the model.
[0015] Step 3: Input the fused visual feature map into the model's BLIP decoder and use the Code Book to generate the quantized image feature map.
[0016] The specific operation of step 3 is as follows:
[0017] Step 3.1: Input the quantized visual feature map into the BLIP decoder of the model, introduce auxiliary task generation, generate a reconstructed image from the input image through the cross attention layer and the Transformer layer, output the text description corresponding to the image, and convert it into a preference distribution of the corresponding vocabulary size;
[0018] Step 3.2: Input the text description from Step 1 into the model's Bert Tokenizer to obtain the corresponding token sequence;
[0019] Step 3.3: Compare the preference distribution output in Step 3.1 with the token sequence in Step 3.2, and calculate the difference between the two using the cross-entropy loss function below:
[0020]
[0021] in, Represents the probability distribution predicted by the model. Distribution of the true target The cross-entropy between the model output and the true label is used to measure the difference between the model output and the true label. Indicates the length of the token sequence. Indicates vocabulary size, Indicates the target distribution. Represents the probability distribution after Softmax transformation:
[0022] Step 3.4: Input the text description from Step 1 into the text encoder of the model to obtain the text feature vector in continuous space;
[0023] Step 3.5: Compare the text feature vector from Step 3.4 with the quantized visual features from Step 2.3 and calculate the loss.
[0024] Step 3.6: Use the visual decoder in the model to convert the quantized visual features obtained in step 2.3 into a reconstructed image, and calculate the L1 loss and mean squared error loss with the input image in step 1;
[0025]
[0026] in, This represents the mean absolute error between the reconstructed image and the original image, and is used to measure the deviation between the two at the pixel level. This represents the total number of pixels in the image. Indicates the first in the reconstructed image The value of each pixel. Indicates the first line in the original graph. The value of each pixel. This represents the absolute difference between corresponding pixels.
[0027]
[0028] in, This represents the mean squared error between the reconstructed image and the original image. It measures the degree of deviation between the two at the pixel level and is more sensitive to larger reconstruction errors. This represents the total number of pixels in the image. Indicates the first in the reconstructed image The value of each pixel. Indicates the first line in the original graph. The value of each pixel. This represents the absolute difference between corresponding pixel values.
[0029] Step 3.7: Feed the reconstructed image from Step 3.6 and the input image from Step 1 into the VGG16 network of the model and calculate the perceptual loss;
[0030]
[0031] in, This represents the average squared error between the reconstructed image and the original image in the feature spaces of different layers of the VGG16 network, used to measure the degree of deviation between the two in high-level semantic features. Indicates the VGG16 network's... Feature representation extracted from layers, These represent the number of channels, width, and height of the feature map in this layer, respectively. Indicates the reconstructed image. This represents the original image. This represents the squared error between corresponding feature points in the reconstructed image and the original image;
[0032] Step 3.8: Treat the entire network as a generative adversarial network and introduce the discriminator DINO v2 to train the generative adversarial network. Use the text decoder to generate scene descriptions of the images, use the visual decoder to reconstruct the input images, use the discriminator network to discriminate the reconstructed images, and output the reconstructed images.
[0033] The specific operation of the auxiliary task in step 3.1 is as follows: the BLIP decoder provides fine-grained information through the input visual feature map and compares it with the description in the real label. Then, the cross-entropy loss is calculated using the formula in step 3.3. The auxiliary task provides a branch supervision signal, and the cross-entropy loss is the branch supervision signal.
[0034] Compared with the prior art, the present invention has the following advantages:
[0035] This invention proposes using text modalities to supplement and enhance the image information contained in discrete tokens, thereby improving the representational capability of the tokenizer. A BLIP decoder with a cross-attention layer is introduced, and an additional auxiliary task is set: this decoder outputs a corresponding scene description using discrete image tokens. Through this joint task, the tokenizer's ability to preserve image details is effectively enhanced, resulting in a higher-performance tokenizer. Subsequent experimental results also verify the effectiveness of this method.
[0036] Introducing additional auxiliary training tasks into the traditional architecture forms branch supervision signals, guiding the model to better understand and retain fine-grained information in images during training, thereby achieving better performance in reconstruction tasks. By introducing additional auxiliary tasks, the representational power of the visual encoder can be effectively enhanced with only a small amount of training data. Simultaneously, using discretized image representation instead of traditional continuous representation significantly improves efficiency in both training and inference phases, achieving the goal of cost reduction and efficiency improvement. Attached Figure Description
[0037] Figure 1 To train the overall flowchart;
[0038] Figure 2 This diagram illustrates the impact of auxiliary tasks on generation quality under different data volumes. Detailed Implementation
[0039] To gain a deeper understanding of this invention, we will provide a comprehensive and detailed description. However, this invention has various implementations and is not limited to the specific examples listed herein. These examples are presented to enhance a full understanding of the disclosure of this invention.
[0040] The dataset used in this invention includes images from everyday life scenarios and their corresponding text descriptions. For example... Figure 1 As shown, the entire program is divided into a training part and a testing part. The training part uses everyday images and their corresponding descriptions, while the testing part uses images from the ImageNet10k dataset.
[0041] A low-cost, high-precision image modeling and reconstruction method, the method comprising the following steps:
[0042] Step 1: Collect an image dataset, filter and input images and their corresponding text descriptions from the dataset, and preprocess the input images to a uniform size;
[0043] Step 2: Construct a low-cost, high-precision image modeling and reconstruction model, use the visual encoder in the model to extract features from the preprocessed image to obtain visual feature maps in continuous space, and then fuse the visual feature maps;
[0044] The specific operation of step 2 is as follows:
[0045] Step 2.1: Process the visual feature map through the attention projection layer in the model, fuse global context information, and obtain a continuous spatial feature map;
[0046] Step 2.2: Compress the continuous spatial feature map obtained in Step 2.1 using a vector compression method to obtain a discrete vector with the closest cosine similarity to the continuous vector, thereby realizing the quantization of the visual feature map;
[0047] Step 2.3: Perform overall information fusion on the quantized visual feature map through another attention projection layer in the model.
[0048] Step 3: Input the fused visual feature map into the model's BLIP decoder and use the Code Book to generate the quantized image feature map.
[0049] The specific operation of step 3 is as follows:
[0050] Step 3.1: Input the quantized visual feature map into the BLIP decoder of the model, introduce auxiliary task generation, generate a reconstructed image from the input image through the cross attention layer and the Transformer layer, output the text description corresponding to the image, and convert it into a preference distribution of the corresponding vocabulary size;
[0051] Step 3.2: Input the text description from Step 1 into the model's Bert Tokenizer to obtain the corresponding token sequence;
[0052] Step 3.3: Compare the preference distribution output in Step 3.1 with the token sequence in Step 3.2, and calculate the difference between the two using the cross-entropy loss function below:
[0053]
[0054] in, Represents the probability distribution predicted by the model. Distribution of the true target The cross-entropy between the model output and the true label is used to measure the difference between the model output and the true label. Indicates the length of the token sequence. Indicates vocabulary size, Indicates the target distribution. Represents the probability distribution after Softmax transformation:
[0055] Step 3.4: Input the text description from Step 1 into the text encoder of the model to obtain the text feature vector in continuous space;
[0056] Step 3.5: Compare the text feature vector from Step 3.4 with the quantized visual features from Step 2.3 and calculate the loss.
[0057] Step 3.6: Use the visual decoder in the model to convert the quantized visual features obtained in step 2.3 into a reconstructed image, and calculate the L1 loss and mean squared error loss with the input image in step 1;
[0058]
[0059] in, This represents the mean absolute error between the reconstructed image and the original image, and is used to measure the deviation between the two at the pixel level. This represents the total number of pixels in the image. Indicates the first in the reconstructed image The value of each pixel. Indicates the first line in the original graph. The value of each pixel. This represents the absolute difference between corresponding pixels.
[0060]
[0061] in, This represents the mean squared error between the reconstructed image and the original image. It measures the degree of deviation between the two at the pixel level and is more sensitive to larger reconstruction errors. This represents the total number of pixels in the image. Indicates the first in the reconstructed image The value of each pixel. Indicates the first line in the original graph. The value of each pixel. This represents the absolute difference between corresponding pixel values.
[0062] Step 3.7: Feed the reconstructed image from Step 3.6 and the input image from Step 1 into the VGG16 network of the model and calculate the perceptual loss;
[0063]
[0064] in, This represents the average squared error between the reconstructed image and the original image in the feature spaces of different layers of the VGG16 network, used to measure the degree of deviation between the two in high-level semantic features. Indicates the VGG16 network's... Feature representation extracted from layers, These represent the number of channels, width, and height of the feature map in this layer, respectively. Indicates the reconstructed image. This represents the original image. This represents the squared error between corresponding feature points in the reconstructed image and the original image;
[0065] Step 3.8: Treat the entire network as a generative adversarial network and introduce the discriminator DINO v2 to train the generative adversarial network. Use the text decoder to generate scene descriptions of the images, use the visual decoder to reconstruct the input images, use the discriminator network to discriminate the reconstructed images, and output the reconstructed images.
[0066] The specific operation of the auxiliary task in step 3.1 is as follows: the BLIP decoder provides fine-grained information through the input visual feature map and compares it with the description in the real label. Then, the cross-entropy loss is calculated using the formula in step 3.3. The auxiliary task provides a branch supervision signal, and the cross-entropy loss is the branch supervision signal.
[0067] In the experimental setup, except for the introduction of the BLIP decoder, the model structure, learning rate, and dataset settings remained consistent with the comparison method. The comparison method removed the attention layer, BLIP decoder, and related auxiliary tasks from the model. The test metric used was Fréchet Inception Distance (FID). The experimental results are as follows: Figure 2 As shown, the results indicate that the present invention achieves higher generation performance with less training data while maintaining the same training speed as the original model, thereby effectively reducing training costs and enhancing the practical value of the model.
[0068] Contents not described in detail in this specification are prior art known to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications are readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of this invention are protected.
Claims
1. A low-cost, high-precision image modeling and reconstruction method, characterized in that, The method includes the following steps: Step 1: Collect an image dataset, filter and input images and their corresponding text descriptions from the dataset, and preprocess the input images to a uniform size; Step 2: Construct a low-cost, high-precision image modeling and reconstruction model, use the visual encoder in the model to extract features from the preprocessed image to obtain visual feature maps in continuous space, and then fuse the visual feature maps; Step 3: Input the fused visual feature map into the model's BLIP decoder and use the Code Book to generate the quantized image feature map.
2. The low-cost, high-precision image modeling and reconstruction method according to claim 1, characterized in that, The specific operation of step 2 is as follows: Step 2.1: Process the visual feature map through the attention projection layer in the model, fuse global context information, and obtain a continuous spatial feature map; Step 2.2: Compress the continuous spatial feature map obtained in Step 2.1 using a vector compression method to obtain a discrete vector with the closest cosine similarity to the continuous vector, thereby realizing the quantization of the visual feature map; Step 2.3: Perform overall information fusion on the quantized visual feature map through another attention projection layer in the model.
3. The low-cost, high-precision image modeling and reconstruction method according to claim 2, characterized in that, The specific operation of step 3 is as follows: Step 3.1: Input the quantized visual feature map into the BLIP decoder of the model, introduce auxiliary task generation, generate a reconstructed image from the input image through the cross attention layer and the Transformer layer, output the text description corresponding to the image, and convert it into a preference distribution of the corresponding vocabulary size; Step 3.2: Input the text description from Step 1 into the model's Bert Tokenizer to obtain the corresponding token sequence; Step 3.3: Compare the preference distribution output in Step 3.1 with the token sequence in Step 3.2, and calculate the difference between the two using the cross-entropy loss function below: in, Represents the probability distribution predicted by the model. Distribution of the true target The cross-entropy between the model output and the true label is used to measure the difference between the model output and the true label. Indicates the length of the token sequence. Indicates vocabulary size, Indicates the target distribution. Represents the probability distribution after Softmax transformation: Step 3.4: Input the text description from Step 1 into the text encoder of the model to obtain the text feature vector in continuous space; Step 3.5: Compare the text feature vector from Step 3.4 with the quantized visual features from Step 2.3 and calculate the loss. Step 3.6: Use the visual decoder in the model to convert the quantized visual features obtained in step 2.3 into a reconstructed image, and calculate the L1 loss and mean squared error loss with the input image in step 1; in, This represents the mean absolute error between the reconstructed image and the original image, and is used to measure the deviation between the two at the pixel level. This represents the total number of pixels in the image. Indicates the first in the reconstructed image The value of each pixel. Indicates the first line in the original graph. The value of each pixel. This represents the absolute difference between corresponding pixels. in, This represents the mean squared error between the reconstructed image and the original image. It measures the degree of deviation between the two at the pixel level and is more sensitive to larger reconstruction errors. This represents the total number of pixels in the image. Indicates the first in the reconstructed image The value of each pixel. Indicates the first line in the original graph. The value of each pixel. This represents the absolute difference between corresponding pixel values. Step 3.7: Feed the reconstructed image from Step 3.6 and the input image from Step 1 into the VGG16 network of the model and calculate the perceptual loss; in, This represents the average squared error between the reconstructed image and the original image in the feature spaces of different layers of the VGG16 network, used to measure the degree of deviation between the two in high-level semantic features. Indicates the VGG16 network's... Feature representation extracted from layers, These represent the number of channels, width, and height of the feature map in this layer, respectively. Indicates the reconstructed image. This represents the original image. This represents the squared error between corresponding feature points in the reconstructed image and the original image; Step 3.8: Treat the entire network as a generative adversarial network and introduce the discriminator DINO v2 to train the generative adversarial network. Use the text decoder to generate scene descriptions of the images, use the visual decoder to reconstruct the input images, use the discriminator network to discriminate the reconstructed images, and output the reconstructed images.
4. The low-cost, high-precision image modeling and reconstruction method according to claim 3, characterized in that, The specific operation of the auxiliary task in step 3.1 is as follows: the BLIP decoder provides fine-grained information through the input visual feature map and compares it with the description in the real label. Then, the cross-entropy loss is calculated using the formula in step 3.
3. The auxiliary task provides a branch supervision signal, and the cross-entropy loss is the branch supervision signal.