Consistency multi-view image generation method and system based on potential diffusion model

Abstract

The invention discloses a consistency multi-view image generation method and system based on a potential diffusion model. The method comprises the following steps: constructing a multi-angle main body consistency data set; a pre-trained potential diffusion model is used as a base model, low-rank adaptation fine tuning training is carried out, and a low-rank adaptation model is obtained; based on a pre-trained potential diffusion model, loading a low-rank adaptation model, inputting a control angle picture or text and a target angle set, and generating a corresponding multi-angle consistency image; performing super-resolution reconstruction on the generated image; cutting: automatically identifying the multi-angle jigsaw puzzle and cutting the multi-angle jigsaw puzzle into a plurality of single-angle images; the cross-view consistency of the generation model can be improved through LoRA (low-rank adaptation) fine tuning and multi-view consistency regularization under the conditions of small samples and low computing power, the stability of the main body is kept through seed sharing and synchronous denoising control in the reasoning stage, the super-resolution and automatic cutting post-processing are combined, and the accuracy of the generation model is improved. And integrated generation and standardized output of multi-angle images are realized.

Description

TECHNICAL FIELD

[0002] The present application relates to the technical field of computer vision and generative artificial intelligence, and in particular to a consistency multi-view image generation method and system based on a latent diffusion model. BACKGROUND

[0004] In the scenarios of building visualization, industrial product design review, marketing e-commerce, etc., it is necessary to generate consistent images of multiple views for the same subject. At present, the mainstream existing technical solutions for realizing multi-view consistent image generation mainly include the following categories:

[0005] 1) The scheme of traditional three-dimensional modeling + rendering process: manually establish the three-dimensional model geometry of the target subject and set its material, set the view angle of rendering, configure the light environment and surrounding environment, output images through raster / ray tracing rendering, and generate multiple images of different views according to different view angles. However, this scheme has long cost chain and low efficiency; it requires high-quality 3D modeling and material and performs multi-camera rendering; it consumes a lot of manpower and computing power, has a long delivery cycle, and is not suitable for fast multi-subject / multi-batch applications.

[0006] 2) The scheme based on per-object fine-tuning: using a pre-trained text-to-image diffusion model (such as Stable Diffusion), a large amount of multi-angle picture data of a specific object is collected, and the pre-trained model is fine-tuned and trained. After fine-tuning, the model can generate images of the specific object at different view angles. Its core is a fine-tuned diffusion model, the input is usually a text prompt or noise, and the output is a single image. If multiple views are needed, multiple generations are needed and the view angle is controlled by the prompt. However, this scheme has the following shortcomings or deficiencies: 1. Low efficiency and unable to generalize: each new object needs to collect a large amount of data and perform a complete fine-tuning training, which is tedious, time-consuming and labor-intensive, and cannot realize fast and generalized generation of new objects. 2. Consistency depends on the prompt: multiple generations depend on accurate view angle prompts, and the internal mechanism of the model does not directly guarantee consistency, which is prone to deviations in appearance, color and details.

[0007] 3) Integration of explicit three-dimensional representation (3D Representation) scheme: combine three-dimensional reconstruction models such as Neural Radiance Field (NeRF), 3D Gaussian Splatting, etc. with diffusion models. Usually, an initial view is generated by the diffusion model first, and then an optimization process is used to infer other views based on the initial view through the three-dimensional reconstruction model, or the diffusion model is used as a prior to optimize the three-dimensional representation. However, this scheme has the following shortcomings or deficiencies: 1. High computational cost: whether based on NeRF optimization or using diffusion model to optimize 3D representation, a long iterative optimization process is required (usually several minutes or even several hours), which cannot achieve real-time or fast generation. 2. System complexity: the need to integrate two different models and technologies, complex system architecture, stability and reliability challenges.

[0008] 4) Full-parameter training scheme based on latent diffusion model (LDM) framework: adjust the basic algorithms of VAE encoder / decoder, text encoder and UNet denoising network within the latent diffusion model (LDM) framework to improve the conditional alignment capability, and conduct full-parameter training based on the new latent diffusion model framework to realize multi-view consistency. However, this scheme has the following shortcomings or deficiencies: 1. High training / computing power cost and slow iteration: VAE, text encoder and UNet are all involved in updating, with large parameter quantity, requiring long-period distributed training and high memory devices. 2. High single adaptation cost; difficult to support multi-subject / multi-batch rapid iteration online. 3. Poor reproducibility and complex maintenance: structural adjustments such as noise scheduling, attention depth, channel width, conditional branch are often made to adapt to data distribution. Different versions of models are not compatible, and the inference / deployment link is frequently changed. 4. Prone to catastrophic forgetting and representation drift: repeated full-parameter fine-tuning of different subjects or new tasks on the same base, bottom features are covered. Old task quality degradation, need to fork multiple large weights, leading to weight volume expansion and complex version management. 5. High acquisition / annotation cost and high generalization risk: data governance and annotation cost increase; once coverage is insufficient, the stability of the model to unseen views decreases.

[0009] 5) The solution of Video Generation Model Adaptation: The multi-view image sequence is regarded as a video frame sequence, and the temporal attention mechanism of the video diffusion model is used to ensure the consistency between frames (i.e. between views). The content and view changes are controlled by inputting prompt words. However, this solution has the following shortcomings or deficiencies: 1. Weak geometric consistency: The temporal attention mechanism of the video model focuses more on the smooth changes in time sequence rather than strict geometric consistency. When the baseline between views is large and the appearance changes dramatically, the problem of body structure distortion or shape inconsistency is easily caused. 2. Inaccurate view control: The control of the generated view is not as accurate and stable as the model designed specifically for multi-view.

[0010] The purpose of the present application is to overcome the shortcomings of the prior art and provide a consistency multi-view image generation method and system based on a diffusion model. The method can improve the cross-view consistency of the generation model under the conditions of small samples and low computing power through LoRA (Low Rank Adaptation) fine-tuning and multi-view consistency regularization, and can maintain the stability of the subject in the inference stage by sharing seeds and synchronous denoising control. Combined with super-resolution and automatic cropping post-processing, the method realizes the integrated generation and standardized output of multi-angle images.

[0011] The information disclosed in this BACKGROUND section is only intended to enhance the understanding of the general background of the application, and should not be considered as recognition or implicit acknowledgment in any form that this information constitutes prior art known to those skilled in the art. SUMMARY

[0013] The purpose of the present application is to provide a consistency multi-view image generation method and system based on a latent diffusion model, which aims to solve the technical problems of low generation efficiency, inability to generalize to new objects, high computational cost, complex system, weak geometric consistency and other technical problems in the prior art. The present application enables the model to quickly and efficiently generate multiple view images of any object that maintains high consistency in appearance, texture and structure without the need for fine-tuning for each new object.

[0014] To achieve the above purpose, the present application adopts the following technical solutions:

[0015] The present application provides a consistency multi-view image generation method based on a latent diffusion model, which includes the following steps:

[0016] S1, construction of a multi-angle subject consistency dataset;

[0017] S2, using a pre-trained latent diffusion model as a base model, performing low-rank adaptive fine-tuning training to obtain a low-rank adaptive model;

[0018] S3, based on the pre-trained latent diffusion model, loading the low-rank adaptive model, inputting the control angle picture or text and the target angle set, and generating the corresponding multi-angle consistency image;

[0019] S4, super-resolution reconstruction is performed on the generated image;

[0020] S5, cutting: automatically identifying and cutting the multi-angle jigsaw into multiple single-angle images.

[0021] Further, the specific implementation process of step S1 is as follows:

[0022] S11, data acquisition and subject grouping: data acquisition and subject grouping are performed on the target object; specifically, different view image samples of the corresponding subject of the building, city landscape and product are collected, and a unique identification ID is assigned to the corresponding subject. Image samples of the same subject under different viewing angles are collected, and each subject is assigned a unique identification ID to facilitate subsequent group sampling and consistency constraint training;

[0023] S12, jigsaw rules and size specifications are formulated, and a jigsaw is constructed accordingly; each subgraph in the jigsaw is a different view image of the same subject;

[0024] S13, the jigsaw data is enhanced to improve the generalization ability of the model; the data enhancement methods include: horizontal flip, random exchange of each sub-view in the layout, or by randomly emptying part of the subgraph area in the jigsaw to simulate the "missing view" situation, thereby enhancing the robustness of the model to incomplete jigsaw; At the same time, the corresponding text annotation is matched;

[0025] S14, text annotation is performed on the jigsaw image: the text label at least contains view quantity information;

[0026] S15, form a complete data set: the training set is a group of samples , wherein is a multi-angle jigsaw, is the corresponding text label; the JSON format metadata information is retained, which records the subject ID, view quantity, jigsaw layout, angle label information, for training and inference reference.

[0027] Further, the specific implementation process of step S2 is as follows:

[0028] S21, load and freeze the pre-trained latent diffusion model;

[0029] The pictures of the data set prepared in step S1 are encoded into latent vectors by VAE ; text description corresponding to each image, converted to conditional vector by text encoder ; since the latent vector of each image is reused in the training process, it is selected to pre-cache these vectors to speed up subsequent iterations

[0030] S23, inject LoRA module:

[0031] Insert LoRA module into cross-attention layers of U-Net down-sampling block, middle block and up-sampling block, and attention layers of text encoder; these attention modules all contain four linear projection matrices: query (Q), key (K), value (V) and output (O); according to the needs, it is selected to insert only linear layer and 1×1 convolution layer; or expand to 3×3 convolution layer on this basis to increase the capturing ability of local patterns of the model; set the dimension of LoRA and scaling factor ; ;

[0032] S24, initialize low-rank matrix; for each inserted layer, split the original weight into fixed base weight W and trainable low-rank update , where , , r is the low-rank dimension, which satisfies ; after initialization, the output weight of each layer modified by LoRA is updated to ;

[0033] S25, training process: follow the training process of diffusion model.

[0034] Further, the specific training process of step S25 is as follows:

[0035] S251, sample noise step, randomly select a diffusion step t from uniform distribution, calculate noise scheduling parameter , the specific calculation method is:

[0036] Set β scheduling table: define a set of variance parameters β1, β2, …, β T increasing with time step t=1,…,T

[0037] Calculate single-step α value: for each step t, calculate: ;

[0038] Here represents the proportion of preserving the original image information in the t-th step

[0039] Calculate cumulative noise scheduling parameter​​​ To directly represent the sample after adding noise at any time step, calculate:

[0040] ;

[0041] S252, generate a noisy latent vector, add noise to the original latent vector , according to the formula:

[0042] ;

[0043] Get the noisy latent vector , where is a standard Gaussian noise;

[0044] S253, denoising prediction:

[0045] Put into the LoRA-injected U-Net to get the predicted noise ;

[0046] represents the noise estimate of the network output with parameters θ;

[0047] The parameter set θ represents all trainable parameters of the neural network;

[0048] S254, loss calculation;

[0049] S255, optimization update;

[0050] S256, loop iteration;

[0051] S257, model output: save the low-rank adaptation weight file periodically during training.

[0052] Further, the specific implementation process of step S3 is as follows:

[0053] Load model: load the pre-trained basic diffusion model and the trained special LoRA weight file;

[0054] Input method A: given a control angle jigsaw containing multiple angles , the jigsaw is composed of subgraphs of the same subject at multiple specified angles, and each subgraph corresponds to an angle label , the model generates a target output corresponding to the input jigsaw in angle according to the text label c:

[0055] ;

[0056] Where G represents the inference function based on the latent diffusion model and LoRA fine-tuning, is the angle the generated image under the following conditions;

[0057] Input mode B: input only text label c, which contains at least the number of perspective information, in this mode, the model directly generates a set of target jigsaw puzzles according to the text condition , without providing control image input.

[0058] Further, the specific implementation process of step S4 is as follows:

[0059] The generated result Apply the upsampling function Perform super-resolution reconstruction.

[0060] Further, the specific implementation process of step S5 is as follows:

[0061] S51, pretreatment and working area purification;

[0062] S52, boundary pre-detection;

[0063] S53, self-estimation of subgraph quantity;

[0064] S54, adaptive cropping;

[0065] S55, consistency score and parameter selection;

[0066] S56, edge refinement and separation zone removal;

[0067] S57, confidence and rollback;

[0068] S58, sequence with super S4: cropping and super-resolution exchange: when the separation zone is obvious, it is recommended to crop first and then super-resolution; When the separation zone is fine / subgraph is small, super-resolution first and then cropping to improve the signal-to-noise ratio.

[0069] A system for implementing a consistency multi-perspective image generation method based on a latent diffusion model, comprising:

[0070] Data processing module: responsible for image standardization and enhancement;

[0071] LoRA training module: responsible for loading the base model, injecting LoRA parameters, and executing the training cycle;

[0072] Inference module: responsible for loading the trained model, receiving user input, executing the generation process and outputting the result;

[0073] Image post-processing module: responsible for automatically identifying and cutting the multi-angle jigsaw puzzle after super-resolution into multiple single-angle images, realizing a set of multiple single-view high-resolution images maintaining multi-perspective consistency.

[0074] By adopting the above technical solution, the present application has the following beneficial effects:

[0075] Low-cost and fast solution to consistency problem: Through the data form of "single image and multiple perspectives" and the training strategy of low-rank adaptation, the model learns the three-dimensional consistency prior in a fast and low-cost manner, and the multiple perspective images generated maintain excellent consistency in appearance, texture and geometric structure; and in the small sample scene, it quickly adapts and significantly reduces the training cost.

[0076] Improve efficiency and generalization ability: The model only needs to be trained once, and can be generalized to countless new objects outside the training set for fast reasoning generation, completely avoiding the tedious process of "one object fine-tuning at a time", and is highly efficient.

[0077] Reduce computational complexity: The entire scheme is based on mature 2D diffusion model architecture, without integrating complex 3D representation and optimization modules, and the inference speed is fast, with significantly lower computational cost than schemes based on NeRF and the like.

[0078] Ensure the quality of the generated images: Based on the powerful generation ability of the diffusion model, the generated images have high fidelity and diversity.

[0079] Flexible application: Supports both unconditional generation and conditional generation modes, and can adapt to the needs of different application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art of the present application, the drawings needed in the description of the specific embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0082] Figure 1 It is a general flowchart of the present application;

[0083] Figure 2 It is a data set construction flowchart of the multi-angle subject consistency of the present application;

[0084] Figure 3 It is a puzzle example schematic diagram of the present application;

[0085] Figure 4 It is a general flowchart of low-rank adaptation fine-tuning training based on a pre-trained latent diffusion model of the present application;

[0086] Figure 5 It is a specific process diagram of low-rank adaptation fine-tuning training based on a pre-trained latent diffusion model of the present application. DETAILED DESCRIPTION

[0088] The technical solutions of the present application will be described clearly and completely in connection with the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0089] The specific embodiments of the present application will be described in detail below in connection with the drawings. It should be understood that the specific embodiments described here are only used to illustrate and explain the present application, and are not used to limit the present application.

[0090] Before further explaining the present application, the technical terms involved in the present application are explained as follows:

[0091] Diffusion Model: An advanced deep generative model. Its working principle is to add noise to the training data step by step through a series of forward steps, and then learn the reverse denoising process, so as to generate new data samples from random noise.

[0092] LDM - Latent Diffusion Model: A model that performs diffusion process in low-dimensional latent space rather than original pixel space, significantly reducing computational cost, and is one of the most popular diffusion model architectures.

[0093] DDPM - Denoising Diffusion Probabilistic Model: A foundational specific implementation form of diffusion model, which explicitly defines its probabilistic modeling framework.

[0094] Masking: In deep learning, it refers to covering a certain part of the input data with a specific value (such as 0 or noise) to prevent the model from seeing that part of the information, which is often used in self-supervised learning training.

[0095] NeRF - Neural Radiance Fields: A technology that represents a three-dimensional scene through a neural network, which can reconstruct a high-quality three-dimensional model from two-dimensional images and generate new perspective images.

[0096] In combination Figures 1-5 As shown in the drawings, the present embodiment provides a consistency multi-view image generation method and system based on diffusion model, and the detailed steps of the method are as follows:

[0097] Step 1, construction of multi-angle subject consistency dataset:

[0098] ​​​​​​​​​​For each different object, obtain images (e.g., front view, side view, back view, top view) taken or rendered from multiple different angles. These images are laid out through a regular grid or irregular grid (e.g., horizontal row, 2x2 grid, etc.) to be spliced into a single image. This spliced image and its corresponding text label serve as a training sample for the model.

[0099] The specific steps are as follows:

[0100] S11, data collection and subject grouping:

[0101] Data collection and subject grouping are performed on the target object. Specifically, image samples of different angles of buildings, cityscapes, products, etc. corresponding subjects are obtained, and a unique identification ID is assigned to the corresponding subject. Image samples of the same subject under different angles are collected, and a unique identification ID is assigned to each subject to facilitate subsequent grouping sampling and consistency constraint training. However, this identification is not a necessary condition, and even if the subject ID is not assigned, the effectiveness of the training can be ensured through the structure of the puzzle itself.

[0102] .

[0103] S12, formulate the puzzle rule and size specification, and construct the puzzle accordingly. The puzzle rule can be a regular layout (such as 2x2, 3x3 grid form), or an irregular layout (such as horizontal row, vertical row, or irregular arrangement). Each sub-picture in the puzzle is a different angle image of the same subject, and the multi-angle puzzle P∈ The row and column spacing and the frame can be 0. To ensure the processing efficiency of the subsequent potential diffusion model, the size of the entire puzzle is uniformly constrained to be an integer multiple of 64 in length and width, such as 512x512 or 768x768 pixels.

[0104] S13, enhance the puzzle data to improve the generalization ability of the model. Data enhancement methods include but are not limited to: horizontal flip, random exchange of each sub-view in the layout, etc. In addition, by randomly emptying part of the sub-picture area in the puzzle (replaced by a uniform fill color or noise), the "missing angle" situation can be simulated, thereby enhancing the robustness of the model to incomplete puzzles. At the same time, the corresponding text label should be matched.

[0105] S14, text labeling of the puzzle image. The text label at least contains the number of views information, such as "a picture of 6 views", and can further contain the subject category (such as building, cityscape, phone, shoes, etc.), or contain specific angle indication (such as front, back, left, right, perspective).

[0106] S15, forming a complete data set. The training set is a set of samples , wherein is a multi-angle puzzle, is a corresponding text label; the metadata information in JSON format is reserved, and the metadata records subject ID, number of perspectives, puzzle layout, angle label, etc. for training and inference reference.

[0107] Step two: use the pre-trained latent diffusion model as a base model for low-rank adaptation fine-tuning training to obtain a low-rank adaptation model:

[0108] S21, loading and freezing the pre-trained latent diffusion model

[0109] A diffusion model pre-trained on a large-scale general image dataset is selected as the base model, such as Stable Diffusion, Latent Diffusion Model (LDM, latent diffusion model), which has powerful image generation and understanding capabilities. Load the variational autoencoder (VAE), text encoder (CLIP) and UNet from the pre-trained latent diffusion model, and fix (freeze) their weights to ensure that they do not update during fine-tuning.

[0110] S22, constructing the latent representation of the training data

[0111] The pictures in the data set prepared in S1 are encoded into latent vectors by VAE ; and the text description corresponding to the image is converted into a conditional vector by the text encoder . Since the latent vector of each picture is reused during training, these vectors can be pre-cached to speed up subsequent iterations.

[0112] S23, inject LoRA module

[0113] Insert the LoRA module into the cross-attention layers of the U-Net downsampling block, the middle block and the upsampling block, and the attention layers of the text encoder. These attention modules all contain four linear projection matrices: query ( ), key ( ), value ( ) and output ( ). According to the needs, you can choose to insert only linear layers and 1x1 convolution layers; or extend to 3x3 convolution layers on this basis to increase the model's ability to capture local patterns. Set the dimensions of LoRA and the scaling factor , for example, set r=32, α=1, which means using a 32-dimensional low-rank matrix.

[0114] S24, initialize low-rank matrix

[0115] For each inserted layer, the original weight is split into a fixed base weight W and a trainable low-rank update where , r is the low-rank dimension, satisfying After initialization, the output weight of each layer rewritten by LoRA is updated as .

[0116] S25, training process

[0117] Training process: follow the training procedure of diffusion model. In the forward process, add noise to the target plane graph step by step; in the backward denoising process, require the model to predict the added noise according to the corresponding data set. By calculating the difference (such as mean square error MSE Loss) between the predicted noise and the real noise as the loss function, the parameters of the LoRA matrix are optimized.

[0118] Specific operation steps:

[0119] S251, sample noise step, randomly select a diffusion step t (for example, between 0-1000) from a uniform distribution, and calculate the noise scheduling parameter .

[0120] The specific calculation method is:

[0121] Set the β scheduling table: define a set of variance parameters β1, β2, …, β T that increase with time step t = 1, …, T.

[0122] (for example, β T increases uniformly from a very small value such as 1 × 10 −4 to a larger value such as 0.02).

[0123] Calculate the single-step α value: for each step t, calculate

[0124] ;

[0125] Here represents the proportion of the original image information retained in the tth step.

[0126] Calculate the cumulative noise scheduling parameter : In order to directly represent the sample after adding noise at any time step, we can calculate:

[0127] ;

[0128] S252, generating a noisy latent vector, to the original latent vector (S22 generated) is added with noise, according to the formula:

[0129]

[0130] The noisy latent vector is obtained , wherein is a standard Gaussian noise.

[0131] S253, denoising prediction

[0132] The is input into the LoRA-injected U-Net to obtain the predicted noise ;

[0133] represents the noise estimate of the network output with parameters θ.

[0134] The parameter set θ represents all trainable parameters of the neural network.

[0135] In a preferred embodiment, the trainable parameters are low-rank matrix parameters inserted by low-rank adaptation (LoRA), and the remaining base model parameters remain frozen; of course, in other embodiments, the parameters θ can also include updateable parameters of the full model or part of the model.

[0136] S254, loss calculation

[0137] The mean square error (MSE) is used as the main training target:

[0138] ;

[0139] wherein, is the Gaussian noise added in step S252, is the LoRA-injected denoising network output result.

[0140] On this basis, in order to ensure the consistency of the same subject under multiple perspectives, the present application can optionally introduce the following regular term:

[0141] Identity consistency loss ;

[0142] Let the frozen identity representation encoder be , and the input is the subgraph , and the output is the instance feature vector:

[0143] ;

[0144] The identity consistency loss is defined as the difference between the feature vectors between different perspectives:

[0145] ;

[0146] Or in cosine similarity form:

[0147] ;

[0148] This loss ensures that the appearance of the same subject generated under different perspectives remains consistent.

[0149] Pose consistency loss :

[0150] When the text annotation contains specific perspective indications (such as front / back / left / right or yaw / pitch / roll values), pose consistency constraints can be introduced:

[0151] ;

[0152] Where Φ is the compliance function between the image and the target pose, used to measure the consistency of the generated subgraph with the specified perspective label, which can be calculated through key point projection error, pose classification confidence, or camera parameter alignment error.

[0153] Local and edge consistency loss , ;

[0154] According to specific application requirements, local key point matching or edge contour detection can also be used to enhance detail and geometric stability.

[0155] Local consistency: Through local key point or local feature matching, the local texture or structure of the same subject under different perspectives is constrained to remain consistent. For example, the logo of a product, the facial features of a person, or the spots of an animal.

[0156] Contour / edge consistency: Through edge detection or contour extraction, the shape of the subject is constrained to maintain geometric reasonableness under different angles. For example, the contour of a building, the body curve of a car, or the body proportion of an animal.

[0157] The final total loss function is:

[0158] ;

[0159] Where, is the basic denoising loss; is the identity consistency loss, ensuring the stability of the subject under different angles; is the pose consistency loss, ensuring that the generated result conforms to the input angle; Optional local / edge consistency regularization for enhancing local details or overall geometric plausibility. , , , are weight coefficients for adjusting the relative importance of each loss term in training.

[0160] Through the above design, the application can not only ensure the basic quality of the generated image, but also maintain the consistency of the subject identity and geometric relationship under different viewing angles, and is suitable for various generation tasks such as buildings, products, characters, animal images, etc.

[0161] S255, optimization update

[0162] Calculate the gradient of the loss with respect to A and B, and update these two low-rank matrices using an appropriate optimizer (such as 8-bit Adam), while all other parameters are not updated, and learning rate scheduling strategies such as cosine annealing restart are adopted.

[0163] If multiple ranks or different levels of learning rates are needed, dynamic low-rank and hierarchical learning rate strategies can be enabled.

[0164] The low-rank dimension (r) is not the higher the better, but needs to be found according to the model, dataset, task, etc. Using dynamic low-rank, multiple dimensions of LoRA can be learned at the same time, thus saving the trouble of finding the best dimension.

[0165] First, select a maximum rank rmax, and then split it into several smaller sub-ranks (such as 4 for each sub-rank). During the training phase, these sub-ranks will participate in the update at the same time, as if multiple LoRA models are trained in parallel. After training is complete, multiple LoRA weight combinations of different ranks can be extracted from this joint model, such as r=4,8,12,16, etc. The optimal rank can be selected according to the validation set performance, without the need for retraining.

[0166] Extended to convolutional layers: Dynamic LoRA is also applicable to 3x3 convolutional layers, where the rank of the convolutional kernel is consistent with the linear layer. Avoid manual parameter tuning and efficiently obtain LoRA adaptation of multiple ranks; especially suitable for scenarios where the sample is limited but the optimal model capacity needs to be found.

[0167] Set different learning rate weights for different layers of U-Net to make the model more flexible when adjusting shallow or deep features.

[0168] Hierarchical learning rate: The structure of U-Net usually includes multiple scale down-sampling blocks, intermediate blocks, and up-sampling blocks, each block representing different levels of features. To make the LoRA adaptation more accurate, different learning rate weights can be set for different blocks:

[0169] Assign weights: Assign weight vectors for the down-sampling part (12 layers), the middle part (1 layer), and the up-sampling part (12 layers), respectively, such as the down-sampling part weight vector is [0.5, 0.5, …, 1.5], the middle sampling part weight vector is 2.0, and the up-sampling part weight vector is [1.5, 1.5, …, 0.5]

[0170] Mechanism: Blocks with high weight values are updated more significantly in training, while blocks with low or even zero weights do not generate LoRA modules or have minimal update amplitude. This allows the model to fully learn new layout patterns at certain scales while maintaining stability at other scales.

[0171] Threshold control: A threshold can be set, and learning rate weights below the threshold will not create LoRA modules, which are used to automatically filter layers that do not need to be adapted.

[0172] Dynamic low rank focuses on adaptively selecting the rank of LoRA, while hierarchical learning rate focuses on applying differentiated updates at different depths of the network. These two strategies can be combined to quickly find the optimal rank and adaptation depth in the case of limited training samples.

[0173] S256, loop iteration

[0174] Repeat steps S252-S254 until a predetermined number of steps is reached, and the training is complete.

[0175] S257, model output

[0176] Periodically save the low-rank adaptation weight file (such as.safetensors format) during training.

[0177] Step three: based on the pre-trained latent diffusion model, load the low-rank adaptation model, input the control angle picture (picture to picture) or text (text to picture) and the target angle set, and generate the corresponding multi-angle consistent image.

[0178] After training, load the low-rank adaptation weight and the base model dynamically, and generate multi-angle consistent images through model inference.

[0179] Load model: Load the pre-trained base diffusion model and the trained special LoRA weight file.

[0180] Input method A (picture to picture): Given a control angle puzzle containing multiple angles , the puzzle is composed of sub-pictures of the same subject at multiple specified angles, and each sub-picture corresponds to an angle label , the model combines the text label c to generate the target output that corresponds to the input puzzle in angle one by one: ; wherein G represents an inference function based on a latent diffusion model and LoRA fine-tuning, for the angle under the generated image. Input mode B (text-to-image): the input is only a text label c, which at least contains the number of view information, and can further contain the subject category information and view indication. In this mode, the model directly generates a set of target jigsaw puzzles according to the text condition without providing control image input.

[0181] In order to ensure the consistency across the views, the present application introduces two key mechanisms in the inference process:

[0182] 1. Shared random seed mechanism: when generating multiple angles, the initial noise of each angle is sampled using the same random seed, so as to ensure that the subject identity and detail features remain stable between different angles.

[0183] 2. Synchronous denoising mechanism: in each step of the diffusion process, the branches of different angles are statistically fused (for example, by mean or attention weighted fusion) in the intermediate feature layer, and then fed back to each angle branch, so as to suppress the geometric drift and identity inconsistency between different views.

[0184] Through the above design, the inference stage of the present application can not only generate consistent images based on multiple angle jigsaw puzzles as input, but also generate multiple angle images only through text labels. No matter which mode is adopted, the generated results can maintain the consistency of the appearance features and geometric relationships of the subject in multiple views, meeting the needs of various application scenarios such as architectural rendering, product effect display, character image synthesis, and animal multi-angle generation.

[0185] Step four: super-resolution reconstruction of the generated image

[0186] The generated result is applied to the upsampling function .

[0187] Preferably, diffusion upsampling: using a latent upsampler of the same system as the base model, enlarging the latent space by ×2 or ×4.

[0188] GAN / reconstruction (optional): ESRGAN / EDSR, etc.

[0189] Step five: automatically identifying and cutting the multi-angle jigsaw puzzles after super-resolution in step four into multiple single-angle images, realizing a set of multiple single-view high-resolution images maintaining multi-view consistency.

[0190] S51, pretreatment and working area purification

[0191] Grayscale and denoise the puzzle image PPP: Apply light Gaussian smoothing (Gaussian σ = 1 ~ 1.5) and Gamma correction (γ ∈ [0.8, 1.2]) to enhance the contrast between the separator and the sub-image interior. Calculate the edge / gradient magnitude (Sobel or Canny) and make one-dimensional row and column projections:

[0192] ;

[0193] Smooth with a window size of 9-15 to get . At the same time, calculate the brightness projection

[0194] ;

[0195] Used to detect "black separators" with low brightness and low variance.

[0196] S52, Boundary pre-detection (candidate tangent set)

[0197] Find local minima (energy valleys) on and , and intersect with low-brightness continuous segments of , to get the vertical candidate tangent set and the horizontal candidate tangent set . Merge adjacent candidates (distance < 8-12 pixels) and extend the line segments to the full height / width. In parallel, use the probabilistic Hough line to detect nearly horizontal / nearly vertical line segments, and project them into , to enhance confidence.

[0198] S53, Sub-image number self-estimation

[0199] The system automatically estimates the number of sub-images through candidate tangents:

[0200] ;

[0201] If is obviously inconsistent with the image structure (see consistency score below), use the alternative estimate:

[0202] Perform connected component analysis (adaptive thresholding, opening and closing operations) on , take the smallest bounding rectangle of the top K areas, merge overlapping and align parallel columns to get ;

[0203] ​​Selecting with consistency score S (see S55) .

[0204] S54, Adaptive cropping

[0205] According to the number and location of the regions directly generate a cropping grid or partition:

[0206] If , directly cut according to the grid;

[0207] If there are only vertical (or horizontal) candidates, first divide the main partition into multiple columns (or rows) according to the (or ), and then in each column (or row) according to the row (or column) energy valley to automatically supplement the secondary cutting line until regions are obtained;

[0208] If there are too many candidate lines, delete them in order from weak to strong according to the "valley depth" and "straight line continuity" until ;

[0209] If there are not enough candidate lines or the separation is very weak, use the connected domain rectangle set to merge into the final cropping region.

[0210] S55, Consistency score and parameter selection

[0211] To ensure the rationality of the cut, a global cost function is defined and minimized:

[0212] ;

[0213] Where is the set of cutting lines adopted; and are the area and aspect ratio variance of all candidate sub-frames, respectively; penalize the misalignment of row and column centers; penalize the cutting line breakage or distortion. The line set and cutting scheme that minimizes J are the results. Typical value ranges: , , , , .

[0214] S56, Edge refinement and separation band removal

[0215] For each cutting line, shrink on both sides according to the normal to remove the black separation band; find the gradient peak in the 5-9 pixel band on each side for 1-2 pixel level "adsorption" fine tuning; perform IoU-NMS on the overlap of adjacent frames to remove duplicates, and perform minimum expansion on small gaps to eliminate gaps. For areas less than The noise frame is removed.

[0216] S57, Confidence and fallback

[0217] Define normalized confidence:

[0218]

[0219] When the number of cut regions is inconsistent with the number of superpixels, automatically perform:

[0220] Threshold adaptive retry (minimum detection threshold, smoothing window, shrinkage ratio);

[0221] Switch "candidate line method connected domain method";

[0222] Output candidate lines and current frame for semi-automatic confirmation.

[0223] S58, Order with super S4

[0224] Cutting and super-resolution can be exchanged: when the separation band is obvious, it is recommended to cut first and then super-resolution; When the separation band is thin / subgraph is small, super-resolution first and then cutting to improve signal-to-noise ratio.

[0225] Brief pseudo code of S5:

[0226] def autocut_no_label(P, p=0.006, tau=0.6):

[0227] G, E = preprocess(P) # gray / denoise / gamma and edge

[0228] Ve, He = project_energy(E) # V_e, H_e

[0229] X = detect_minima(Ve); Y = detect_minima(He)

[0230] X, Y = merge_and_extend(X), merge_and_extend(Y)

[0231] # Number self-estimation (grid and connected domain two choices, determined by consistency score)

[0232] x1 = (len(X)+1)*(len(Y)+1)

[0233] boxes1 = grid_or_fill(X, Y) # Direct cutting or primary and secondary partition fill line

[0234] score1 = consistency_score(boxes1, X, Y)

[0235] boxes2 = cc_boxes(G) # Connected components

[0236] score2 = consistency_score(boxes2, None, None)

[0237] boxes = boxes1 if score1 >= score2 else boxes2

[0238] boxes = refine_boxes(boxes, E, p) # Remove black bands / adsorption / correction

[0239] conf = confidence(boxes, Ve, He)

[0240] return [crop(P, b) for b in boxes], boxes, conf

[0241] The system architecture for implementing the above method may include:

[0242] Data processing module: responsible for image standardization and enhancement;

[0243] LoRA training module: responsible for loading the base model, injecting LoRA parameters, and executing the training loop;

[0244] Inference module: responsible for loading the trained model, receiving user input, executing the generation process, and outputting the results;

[0245] Image post-processing module: Responsible for automatically recognizing and segmenting the multi-angle mosaic after super-resolution into multiple single-angle images, realizing a set of multiple single-view high-resolution images that maintain consistency across multiple viewpoints.

[0246] Compared with existing technologies, the innovative aspects of this application include:

[0247] 1) Collect samples from different subjects and perspectives, create a mosaic of multiple perspectives of the same subject, and generate corresponding text files;

[0248] 2) Perform data augmentation by horizontally flipping the labeled images and randomly swapping sub-viewpoints;

[0249] 3) Insert a LoRA low-rank adaptation module into the pre-trained latent diffusion model, setting the dimensions and scaling factor, using the formula ( ) Adjust the weight, only train the low-rank matrix (A, B);

[0250] 4) Fine-tune the model using mean square error loss or cosine; can be extended to conditional regularization terms, dynamic low-rank adaptation, hierarchical learning rate fine-tuning;

[0251] Conditional regularization can be one or more; Set identity consistency, pose consistency, local consistency and structure constraint, etc. Conditional regularization terms

[0252] 5) Load the LoRA weight obtained by training back to the base model, and generate multi-perspective image sets with subject consistency according to the input text and multi-perspective control map.

[0253] 6) In the post-processing stage, after super-resolution reconstruction of the generated image, crop to realize a set of multi-perspective high-resolution images that maintain multi-perspective consistency, and the order of super-resolution and cropping can be exchanged.

[0254] 7) A system including a data processing module, a LoRA training module, an inference module, and an image post-processing module.

[0255] Preferably, in order to achieve the purpose of the present application, the alternative solutions that can be adopted include:

[0256] Alternative of splicing method: The splicing method of the multi-perspective image is not limited to horizontal or grid arrangement, but can also be any other form of regular or irregular layout, as long as the multi-perspective information can be integrated into an image.

[0257] Alternative of mask strategy: The mask technology is not limited to rectangular mask, but can also be random block mask, irregular shape mask, or semantic mask based on image segmentation, etc., as long as the purpose of forcing the model to learn the perspective inference ability can be achieved.

[0258] Alternative of condition generation input method: In condition generation, the condition information is not limited to image, but can also be image feature vector extracted by encoder, pose (Pose) parameter describing perspective relationship, etc.

[0259] Alternative of model architecture: The diffusion model is not limited to LDM or DDPM, but can also be other types of generation model (such as generation model based on score) or new generation of diffusion model architecture that may appear in the future, as long as it adopts the data construction and training strategy proposed in the present disclosure.

[0260] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A consistent multi-view image generation method based on a latent diffusion model, characterized in that, Includes the following steps: S1. Construction of a dataset with multi-perspective subject consistency; S2. Using the pre-trained latent diffusion model as the base model, perform low-rank adaptation fine-tuning training to obtain the low-rank adaptation model. S3. Based on the pre-trained latent diffusion model, load the low-rank adaptation model, input the control angle image or text and the target angle set, and generate the corresponding multi-angle consistent image. S4. Perform super-resolution reconstruction on the generated image; S5. Cropping: Automatically recognizes and cuts multi-angle mosaics into multiple single-angle images.

2. The consistent multi-view image generation method based on a latent diffusion model according to claim 1, characterized in that, The specific implementation process of step S1 is as follows: S11. Data Acquisition and Subject Grouping: Data acquisition and subject grouping are performed on the target objects. Specifically, image samples from different perspectives of the corresponding subjects of buildings, urban landscapes, and products are collected, and a unique identifier ID is assigned to each subject. Image samples of the same subject from different perspectives are collected, and a unique identifier ID is assigned to each subject to facilitate subsequent group sampling and consistency constraint training. S12. Establish jigsaw puzzle rules and size specifications, and construct the jigsaw puzzle accordingly; each sub-image in the jigsaw puzzle is an image of the same subject from different perspectives; S13. Enhance the jigsaw puzzle data to improve the model's generalization ability; Data augmentation methods include: horizontal flipping, random swapping of subviews in the layout, or simulating a "missing view" situation by randomly emptying some subview areas in the puzzle, thereby enhancing the model's robustness to incomplete puzzles; and matching corresponding text annotations. S14. Add text annotations to the jigsaw puzzle images: The text labels should at least include information on the number of viewpoints; S15. Forming a complete dataset: The training set consists of a set of samples. ,in For multi-angle jigsaw puzzles, The corresponding text labels are provided; metadata information in JSON format is retained, which records the subject ID, number of viewpoints, puzzle layout, and angle label information for training and inference.

3. The consistent multi-view image generation method based on a latent diffusion model according to claim 1, characterized in that, The specific implementation process of step S2 is as follows: S21. Load and freeze the pre-trained latent diffusion model: We selected a pre-trained diffusion model on a large-scale general image dataset as the base model, and loaded variational autoencoder (VAE), text encoder (CLIP) and UNet from the pre-trained latent diffusion model, fixing their weights to ensure that they do not update during fine-tuning. S22. Constructing the latent representation of the training data: The images in the dataset prepared in step S1 are encoded into latent vectors using VAE. The text descriptions corresponding to the images are converted into conditional vectors using a text encoder. Since the latent vectors of each image are reused during training, these vectors are pre-cached to accelerate subsequent iterations. S23, Injecting the LoRA module: The LoRA module is inserted into the cross-attention layers of the U-Net downsampling block, intermediate block, and upsampling block, as well as the attention layer of the text encoder; these attention modules all contain four linear projection matrices: query( ),key( ),value( ) and output ( ); as needed, choose to insert only linear layers and 1×1 convolutional layers; or extend this to 3×3 convolutional layers to enhance the model's ability to capture local patterns; set the dimensions of LoRA. and scaling factor ; S24. Initialize the low-rank matrix: For each insertion layer, the original weights Decomposed into fixed base weights W and trainable low-rank updates. ,in , r is a low-rank dimension, satisfying After initialization, the output weights of each layer rewritten by LoRA are updated to... ; S25. Training process: Following the training process of the diffusion model, noise is gradually added to the target plane map during the forward pass; during the backward denoising process, the model is required to predict the added noise based on the corresponding dataset, and the parameters of the LoRA matrix are optimized by calculating the difference between the predicted noise and the real noise as the loss function.

4. The consistent multi-view image generation method based on the latent diffusion model according to claim 3, characterized in that, The specific training process for step S25 is as follows: S251. Sampling noise step: Randomly select a diffusion step t from the uniform distribution and calculate the noise scheduling parameters. The specific calculation method is as follows: Define a β scheduling table: Define a set of variance parameters β1, β2, ..., β that increase with time steps t=1,…,T. T ; Calculate the single-step α value: Calculate for each step t: ; here This represents the proportion of the original image information retained in step t; Calculate cumulative noise scheduling parameters To directly represent the sample with added noise at any time step, calculate: ； S252. Generate a noisy latent vector and input it into the original latent vector. Add noise to it, according to the formula: ； Obtain the noisy latent vector ,in It is standard Gaussian noise; S253, Denoising Prediction: Will The input is injected into a LoRA-based U-Net, resulting in predicted noise. ; This represents "noise estimation of the network output with parameter θ"; The parameter set θ represents all trainable parameters of the neural network; S254, Loss Calculation: Using mean squared error (MSE) as the primary training objective: ； in, The Gaussian noise added in step S252, The output of the denoising network with LoRA is shown. S255, Optimization and Update: Calculate the gradient of the loss with respect to A and B, and update these two low-rank matrices using an appropriate optimizer, while all other parameters are not involved in the update, and implement a learning rate scheduling strategy. S256, Iterative Loop: Repeat steps S252-S255 until the predetermined number of steps is reached to complete the training. S257, Model Output: During training, the low-rank adaptation weight file is periodically saved.

5. The consistent multi-view image generation method based on a latent diffusion model according to claim 1, characterized in that, The specific implementation process of step S3 is as follows: Load Model: Load the pre-trained base diffusion model and the pre-trained LoRA weight file; Input Method A: Given a jigsaw puzzle containing multiple control angles. The jigsaw puzzle is composed of sub-images of the same subject viewed from multiple specified angles, with each sub-image corresponding to an angle label. The model combines the text label 'c' to generate a target output that corresponds one-to-one with the input puzzle piece in terms of angle: ； Where G represents the inference function based on the latent diffusion model and LoRA fine-tuning, For angle The generated image; Input Method B: The input is only a text label c, which contains at least information about the number of viewpoints. In this mode, the model directly generates a set of target puzzle pieces based on the text conditions. No control image input is required.

6. The consistent multi-view image generation method based on a latent diffusion model according to claim 1, characterized in that, The specific implementation process of step S4 is as follows: For the generated results Apply upsampling function Perform super-resolution reconstruction.

7. The consistent multi-view image generation method based on a latent diffusion model according to claim 1, characterized in that, The specific implementation process of step S5 is as follows: S51. Pretreatment and purification in the working area: Convert the jigsaw puzzle image to grayscale using PPP and reduce noise: Then, apply a light smoothing and Gamma correction to improve the contrast between the separator bands and the interior of the subgraphs; calculate the edge / gradient magnitude for G. And perform one-dimensional row and column projections: ； right Smooth the window size to 9–15 to obtain... Simultaneously calculate the brightness projection: ； Used to detect "black dividing bands" with low brightness and low variance; S52, Boundary Pre-detection: exist and Find the local minimum and compare it with , Intersecting the low-brightness continuous segments yields a set of vertical candidate tangents. With the set of horizontal candidate tangents Merge nearest neighbor candidates and extend the line segment to full height / width; in parallel, use probabilistic Hough lines. Detect near-horizontal / near-vertical line segments and project them onto... , To enhance confidence; S53, Subgraph Quantity Self-Estimation: The system automatically estimates the number of subgraphs based on candidate tangents: ； like If the estimation method is clearly inconsistent with the image structure, then the alternative estimation method is used: right Perform connected component analysis, select the smallest bounding rectangle with the largest area (K), merge overlapping and parallel columns to obtain the result. ; Selected by consistency score S ; S54, Adaptive cropping: according to The quantity and position directly generate the clipping mesh or partitions: like If so, then directly divide according to the grid; If only vertical or horizontal candidates are available, first press or The main segment is divided into multiple columns or rows, and then secondary tangents are automatically added within each column or row based on the energy valleys of the rows or columns until a result is obtained. One region; If there are too many candidate lines, delete them in order of increasing "valley depth" and "straight-line continuity" until the desired result is obtained. ; If there are insufficient candidate lines or the separation is extremely weak, use the set of connected rectangles as the final clipping region; S55. Consistency Score and Parameter Selection: To ensure the rationality of the partitioning, a global cost function is defined and minimized: ； in The set of tangents used; and These represent the area and aspect ratio variances of all candidate subframes, respectively. Penalize rows and columns that are not aligned with their centers; Penalize tangent breakage or distortion; the result is the set of lines and the splitting scheme that minimizes J. S56. Edge Refinement and Separator Removal: For each tangent, reduce inwards on both sides according to the normal direction. Remove black dividing bands; find gradient peaks within 5–9 pixels of each edge and perform 1–2 pixel-level "snap" fine-tuning; perform IoU-NMS deduplication on overlapping adjacent boxes, and perform minimum expansion on small gaps to eliminate voids; for areas smaller than The noise frames are removed. S57. Confidence Level and Backoff: Define normalized confidence level: ； when Or the number of cutting areas and If there is a discrepancy, execute automatically: Threshold-adaptive retry; Switch to "candidate line method" "Connected Component Method"; Output candidate lines and the current box for semi-automatic confirmation; The order of S58 and super-resolution S4: Crop and super-resolution swap: When the dividing bands are obvious, it is recommended to crop first and then super-resolution; when the dividing bands are thin or the sub-images are small, super-resolution should be performed first and then cropping to improve the signal-to-noise ratio.

8. A system for implementing the consistent multi-view image generation method based on the latent diffusion model as described in any one of claims 1-7, characterized in that, include: Data processing module: responsible for image standardization and enhancement; LoRA training module: responsible for loading the base model, injecting LoRA parameters, and executing the training loop; Inference module: responsible for loading the trained model, receiving user input, executing the generation process, and outputting the results; Image post-processing module: Responsible for automatically recognizing and segmenting the multi-angle mosaic after super-resolution into multiple single-angle images, realizing a set of multiple single-view high-resolution images that maintain consistency across multiple viewpoints.

Images