Image generation method and related device

Abstract

The invention discloses an image generation method and a related device, and relates to the field of automobiles, and the method comprises the steps: carrying out the coding of a first image and first shape information, so as to obtain a first vector; at least transmitting a noise image and the first vector to a target prediction model to obtain a denoising prediction result obtained by the target prediction model according to the first shape information and the second shape information; the second shape information is the same as or different from the first shape information; and decoding the de-noising prediction result to obtain a second image, the second image having the second shape information.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image generation method and related apparatus. Background Technology

[0002] Currently, when using machine learning models for extension drawing, i.e., image expansion, the expanded image will have the same shape as the input image because the model does not pay attention to the image shape.

[0003] To obtain images of different shapes, the machine learning model needs to be fine-tuned and retrained, which results in high implementation complexity. Summary of the Invention

[0004] In view of the above problems, this application provides an image generation method and related apparatus to achieve controllable shape in image generation. The specific solution is as follows:

[0005] The first aspect of this application provides an image generation method, characterized in that the method includes:

[0006] The first image and the first shape information are encoded to obtain the first vector;

[0007] At least the noisy image and the first vector are transmitted to the target prediction model to obtain the denoising prediction result obtained by the target prediction model based on the first shape information and the second shape information; the second shape information may be the same as or different from the first shape information;

[0008] The denoising prediction result is decoded to obtain a second image, which has the second shape information.

[0009] In one possible implementation, the target prediction model is either a first prediction model or a second prediction model;

[0010] The first prediction model obtains the denoised prediction result after T time steps; T is a positive integer; the second prediction model obtains the denoised prediction result after S time steps; S is a positive integer; the second prediction model is a latent consistency model obtained by distilling the first prediction model;

[0011] Where S is less than T.

[0012] In one possible implementation, the first prediction model is trained in the following manner:

[0013] In the sample image, an anchor view and a target view are cropped; the anchor view has input shape information, and the target view has output shape information;

[0014] The anchor view is encoded based on the input shape information to obtain an input vector;

[0015] Based on the output shape information, the target view with the added noise image is encoded to obtain an output vector;

[0016] The noisy image and the input vector are transmitted to the first prediction model at least to obtain the sample prediction result obtained by the first prediction model after T time steps;

[0017] Based on the sample prediction results and the output vector, the first loss data is obtained;

[0018] Based on the first loss data, the first prediction model is optimized.

[0019] In one possible implementation, the second prediction model is obtained in the following way:

[0020] The model parameters of the first prediction model are imported into a pre-built base model, teacher model, and student model; the base model, teacher model, and student model are all neural network models.

[0021] In the T time steps of the first prediction model, multiple first time steps and a second time step corresponding to each first time step are sampled. The second time step is after the first time step and is separated from the first time step by k time steps; k is a positive integer.

[0022] For each of the first time steps, perform the following processing:

[0023] Using the base model, the teacher model, and the student model, the input data of the first prediction model at the first time step is processed to obtain the first function value corresponding to the first time step and the second function value corresponding to the second time step;

[0024] Based on the first function value and the second function value, the second loss data is obtained;

[0025] Based on the second loss data, the base model is optimized to obtain the second prediction model.

[0026] In one possible implementation, the second noise sample includes: noise samples with added control elements and noise samples without added control elements; the control element is: the input data of the first prediction model at the first time step in the T time steps;

[0027] The second noise sample is predicted using the remaining time step to obtain a second prediction result, including:

[0028] The noise samples with and without control elements in the second noise sample are predicted using the remaining time steps to obtain the second prediction result.

[0029] In one possible implementation, after decoding the denoised prediction result to obtain a second image, the method further includes:

[0030] The second image is overlaid onto the three-dimensional unfolded image of the target object at at least one calibrated position.

[0031] In one possible implementation, at least the noisy image and the first vector are transmitted to the target prediction model to obtain the denoised prediction result obtained by the target prediction model based on the second shape information, including:

[0032] The noisy image, the first vector, and the relative position vector are transmitted to the target prediction model to obtain the denoising prediction result obtained by the target prediction model based on the second shape information and the relative position vector;

[0033] The second image and the first image are in a relative positional relationship corresponding to the relative position vector.

[0034] A second aspect of this application provides an image generation apparatus, comprising:

[0035] A vector encoding unit is used to encode the first image and the first shape information to obtain a first vector;

[0036] A denoising prediction unit is configured to transmit at least the noisy image and the first vector to a target prediction model to obtain a denoising prediction result obtained by the target prediction model based on the first shape information and the second shape information; the second shape information may be the same as or different from the first shape information.

[0037] An image decoding unit is used to decode the denoising prediction result to obtain a second image, the second image having the second shape information.

[0038] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the image generation method described in the first aspect or any implementation thereof.

[0039] A fourth aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0040] The memory is used to store computer programs;

[0041] The processor is used to execute the computer program so that the electronic device can implement the image generation method of the first aspect or any implementation thereof.

[0042] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to perform the image generation method described in the first aspect or any implementation thereof.

[0043] By employing the above technical solution, the image generation method and related apparatus provided in this application first encode the first image and the first shape information. This allows the obtained first vector and the noisy image to be transmitted to the target prediction model. The target prediction model then obtains a denoising prediction result based on the first shape information and second shape information that is the same as or different from the first shape information. After decoding, the second image corresponding to the second shape information is obtained. Therefore, in this application, the first shape information of the first image is included in the image encoding process. This allows the target prediction model to output a corresponding denoising prediction result based on the first shape information and the second shape information input into the model. After decoding, the image corresponding to the second shape information is obtained, thereby achieving shape-controllable image generation. Attached Figure Description

[0044] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0045] Figure 1 A flowchart illustrating an image generation method provided in this application embodiment;

[0046] Figure 2 This is a flowchart illustrating the training of the first prediction model in an embodiment of this application;

[0047] Figure 3 This is an example diagram showing the training input and output of the UViT model in the embodiments of this application;

[0048] Figure 4 The flowchart below shows the process of distilling the first prediction model to obtain the second prediction model in an embodiment of this application.

[0049] Figure 5 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application;

[0050] Figure 6This is another schematic diagram of an image generation apparatus provided in an embodiment of this application;

[0051] Figure 7 This is another schematic diagram of an image generation apparatus provided in an embodiment of this application;

[0052] Figure 8 This is another schematic diagram of an image generation apparatus provided in an embodiment of this application;

[0053] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0054] Figure 10 This is a schematic diagram illustrating model training and distillation in an extended scenario applicable to car cover background images, as described in this application. Detailed Implementation

[0055] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0056] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0057] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0058] Currently, in the field of image extension rendering, images are mostly generated based on diffusion models combined with position query mechanisms. This approach typically involves using relative position embeddings (RPEs) to effectively control the context of image rendering and to gradually remove noise by predicting it through the model. The advantage of this method is that the content of the extended rendering portion of the image is harmonious and complete, but its shortcomings are equally obvious.

[0059] The main defects are as follows:

[0060] 1. Fixed shape of generated image: Current mainstream research basically controls the shape of the input image to a fixed value. Since the diffusion model does not care about the shape, the shape of the result after expanding the image will be the same as the shape of the input image. To obtain images with different pixel shapes, the model needs to be fine-tuned and retrained, which is very tedious.

[0061] 2. Slower image generation speed: The most advanced model in the field of image extension rendering is PQDiff, which uses the DPM-Solver method for inference. Compared with the latent-consistency-models (LCM) proposed in current diffusion models, it requires more iterations. Moreover, under low iteration conditions, DPM-Solver's denoising effect is worse than LCM with the same number of iterations.

[0062] 3. Image quality issues after image expansion: This is a derivative of the first problem. Because the pixels of the input and output images are fixed and the same, but after image expansion, the content density of the output image will be greater than that of the input image, resulting in poor image quality after expansion.

[0063] 4. Semantic deviation in image content after high-magnification image expansion: Most image extension rendering models often exhibit deviations in the information conveyed by the image from the original image when the expansion magnification is very high (i.e., when expanding more content based on the anchor view). How to ensure that the semantics of the expanded image remain as consistent as possible with the original image still requires further exploration.

[0064] 5. Challenges in the initialization and refinement stages: Existing methods may not be able to accurately capture all the details in the image when using random cropping in the initialization stage, while in the subsequent refinement stage, over-optimization may lead to unnatural generated content.

[0065] In summary, although some progress has been made in image extension rendering, challenges remain, including fixed pixel shapes in generated images, image quality issues after expansion, slow image generation speed, semantic biases in high-magnification expansion, and problems related to initialization and thinning. These shortcomings limit the widespread application of image extension rendering technology in fields such as virtual reality and automotive design.

[0066] To address the aforementioned problems, this application provides an image generation method. The image generation method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0067] Reference Figure 1 , Figure 1This is a flowchart illustrating an image generation method provided in an embodiment of this application. This method can be applied to electronic devices capable of data processing, such as computers or servers. The technical solution in this embodiment is mainly used to achieve shape controllability in image generation.

[0068] Specifically, the method in this embodiment may include the following steps:

[0069] Step 101: Encode the first image and the first shape information to obtain the first vector.

[0070] In this embodiment, when encoding the first image using an encoder, the first shape information can be added, so that the first image and the first shape information participate in the encoding together to obtain the first vector.

[0071] Step 102: At least the noisy image and the first vector are transmitted to the target prediction model to obtain the denoising prediction result obtained by the target prediction model based on the first shape information and the second shape information.

[0072] The second shape information may be the same as or different from the first shape information. For example, the first shape information is the shape information corresponding to the first image, which is the shape information of 512*512 image pixels; the second shape information is the shape information corresponding to the image to be generated, which is the shape information of 512*512 or 1024*1024 image pixels.

[0073] It should be noted that the second shape information can be pre-input according to the business requirements of image generation. Therefore, after transmitting the noisy image and the first vector to the target prediction model, the target prediction model can obtain the denoising prediction result based on the pre-input second shape information. Different second shape information can be input into the target prediction model under different business requirements.

[0074] For example, the second shape information is a 4096*4096 shape information set by the user according to business requirements. In this way, the target prediction model can obtain a denoising prediction result corresponding to 4096*4096 for the noisy image and the first vector (the first image is encoded with 512*512).

[0075] Specifically, the target prediction model can be the UViT model, which is the U-shaped visual Transformer model.

[0076] Step 103: Decode the denoised prediction result to obtain the second image.

[0077] The second image contains second shape information. Specifically, in this embodiment, a decoder can be used to decode the denoising prediction result to obtain a second image with second shape information.

[0078] It should be noted that, in the training process of the target prediction model, in this embodiment, the output shape information of the image to be generated and the input vector obtained by encoding the input image and its input shape information can be used together as the model input to train the target prediction model. This allows the target prediction model to learn that the output of the input image with input shape information can decode the output image with output shape information and thus, when using the target prediction model to generate images, the shape can be controlled in the image generation by using the first shape information and the input second shape information.

[0079] By employing the above technical solution, the image generation method provided in this application first encodes the first image and the first shape information. This allows the obtained first vector and the noisy image to be transmitted to the target prediction model. The target prediction model then obtains a denoising prediction result based on the first shape information and second shape information that differs from the first shape information. After decoding, the second image corresponding to the second shape information is obtained. Therefore, in this application, the first shape information of the first image is included in the image encoding process. This enables the target prediction model to output a corresponding denoising prediction result based on the first shape information and the second shape information input to the model. After decoding, the image with the corresponding second shape information is obtained, thereby achieving shape-controllable image generation.

[0080] In one implementation, taking image extension rendering (i.e., image expansion) as an example, in step 102, the noisy image, the relative position vector, and the first vector can be transmitted to the target prediction model to obtain the denoising prediction result obtained by the target prediction model based on the second shape information and the relative position vector. The second image decoded by the denoising prediction result has a relative positional relationship with the first image corresponding to the relative position vector. That is, the relative position vector is a vector constructed from the relative positional relationship between the first image and the image to be generated. Thus, the relative position vector input into the target prediction model determines the relative positional relationship between the second image decoded by the denoising prediction result output by the target prediction model and the first image.

[0081] Specifically, in the training process of the target prediction model, this embodiment can also use the relative position vector between the input image and the output image as model input to train the target prediction model. This allows the target prediction model to learn to output a denoised prediction result that decodes an image with a corresponding relative positional relationship to the input image. Therefore, using the target prediction model to expand the image can avoid semantic bias, resulting in a more harmonious image content in the obtained second image.

[0082] For example, taking the scenario of expanding a car cover theme image into a car cover background image as an example, in this embodiment, the relative position vector constructed by the relative position relationship between the car cover background image to be generated and the car cover theme image is used as the input of the target prediction model. At the same time, the first vector obtained by encoding the car cover theme image and the 512*512 shape information (shape information of the car cover theme image) and the noise image are used as the input of the target prediction model. This allows the target prediction model to output the corresponding denoising prediction result based on the car cover theme image according to the relative position vector and the 1024*1024 shape information of the image to be generated (shape information of the image to be generated). After decoding, a car cover background image with a shape of 1024*1024 that satisfies the corresponding relative position relationship with the car cover theme image is obtained. As a result, there will not be too much semantic deviation between the obtained car cover background image and the car cover theme image, and the image content is more harmonious.

[0083] In one implementation, the target prediction model can be either a first prediction model or a second prediction model, wherein:

[0084] The first prediction model obtains a denoised prediction result after T time steps; the second prediction model obtains a denoised prediction result after S time steps. The second prediction model is a latent consistency model obtained by distilling the first prediction model. Here, S and T are both positive integers, and S is less than T.

[0085] In other words, in this embodiment, a trained first prediction model can be used to perform denoising prediction on the first vector for T time steps to obtain a denoising prediction result that can decode the second image. During the T time steps of prediction, the first prediction model uses the denoising result output from the previous time step to predict the denoising result for the next time step, ensuring consistency between the denoising results of adjacent time steps. Furthermore, in this embodiment, after the first prediction model is trained, knowledge distillation can be performed on it. The resulting second prediction model can guarantee consistency between denoising results spaced k time steps apart. Therefore, the time steps required for the second prediction model to obtain the denoising prediction result can be significantly reduced, accelerating the efficiency of obtaining the denoising prediction result and thus improving the image generation rate through model acceleration.

[0086] For example, the first prediction model can be the UViT model, and the second prediction model can be the LCM model obtained by distilling the UViT model.

[0087] In one implementation, the first prediction model can be trained in the following way: Figure 2 As shown:

[0088] Step 201: In the sample image, crop the anchor view and the target view.

[0089] The anchor view contains input shape information, and the target view contains output shape information.

[0090] For example, such as Figure 3 As shown in the figure, taking the UViT model as the first prediction model as an example, the anchor view and the target view are randomly extracted from the sample image.

[0091] Step 202: Encode the anchor view based on the input shape information to obtain the input vector.

[0092] For example, such as Figure 3 As shown, the input shape information of the anchor view is encoded in the first encoder to obtain the input vector.

[0093] Step 203: Encode the target view with added noise based on the output shape information to obtain the output vector.

[0094] The noisy image can be a random noise image. For example, such as... Figure 3 As shown, in the second encoder, the output shape information of the target view with added noise is encoded to obtain the output vector.

[0095] Step 204: Transmit at least the noisy image and the input vector to the first prediction model to obtain the sample prediction results obtained by the first prediction model after T time steps.

[0096] For example, a noisy image and input vector can be fed into the UViT model to obtain the sample prediction results output by the UViT model.

[0097] It should be noted that in the image expansion scenario, before step 204, the relative position vector corresponding to the relative position relationship between the anchor view and the target view in the sample image can be obtained first. In this way, in step 204, the noise image, the relative position vector and the input vector can be transmitted to the first prediction model to obtain the sample prediction result obtained by the first prediction model based on the output shape information after T time steps.

[0098] For example, such as Figure 3As shown, the noisy image, input vector, and relative position vector are transmitted to the UViT model to obtain the sample prediction results output by the UViT model.

[0099] Step 205: Obtain the first loss data based on the sample prediction results and the output vector.

[0100] In this embodiment, a loss function can be used to calculate the loss between the sample prediction result and the output vector to obtain the first loss data.

[0101] For example, the loss function can be a loss function based on mean squared error, or it can be a maximum likelihood loss function, and so on.

[0102] Step 206: Optimize the first prediction model based on the first loss data.

[0103] Specifically, in this embodiment, the model parameters in the first prediction model can be adjusted according to the first loss data. After multiple optimizations, the trained first prediction model can be obtained.

[0104] As can be seen, in this embodiment, input shape information and output shape information are added as model inputs during the training process of the first prediction model, so that the first prediction model learns the shape changes in the image generation process. Specifically, the first prediction model learns the correspondence between input shape information and output shape information, and thus learns that the denoising prediction result of the output image that can decode the output shape information is the input image with input shape information. This achieves controllable shape in the image generation process.

[0105] In one implementation, after the first prediction model is trained, the second prediction model can be obtained in the following way: Figure 4 As shown:

[0106] Step 401: Import the model parameters of the first prediction model into the pre-built base model, teacher model, and student model.

[0107] The base model, teacher model, and student model are all neural network models. The base model can also be called nnet, the teacher model can also be called teacher_nnet, and the student model can also be called target_nnet. These three models are the neural network models used to implement model distillation.

[0108] Specifically, in this embodiment, the model parameters of the first prediction model can be assigned to the base model, the teacher model, and the student model, so that all three models can inherit the model learning ability of the trained first prediction model.

[0109] Step 402: In the T time steps of the first prediction model, sample multiple first time steps and the second time step corresponding to each first time step.

[0110] The second time step follows the first time step and is k time steps apart from the first time step; k is a positive integer. For example, k can be 20. The first time step can also be called the start time step (start_timesteps), and the second time step can also be called the interval time step (timesteps).

[0111] Specifically, in this embodiment, random sampling can be performed in T time steps, and the number of samples can be the batch size to obtain the sequence of the first time step. Then, k is subtracted from each item in the sequence of the first time step to obtain the corresponding second time step.

[0112] In addition, in this embodiment, the boundary scaling coefficients c for the first time step and the second time step are obtained respectively. Here, c includes c_skip and c_out. These two boundary scaling coefficients are used to adjust the proportion of input and network output in the corresponding model.

[0113] For each first time step, perform the following processing:

[0114] Step 403: Using the base model, teacher model, and student model, process the input data of the first prediction model at the first time step to obtain the first function value corresponding to the first time step and the second function value corresponding to the second time step.

[0115] Wherein, the first function value is the loss function value between the first noise sample obtained using the base model and the first prediction result; the second function value is the loss function value between the third noise sample obtained using the student model and the third prediction result; the third noise sample is the noise sample obtained by processing the noise-added data corresponding to the first time step obtained by the teacher model using the student model.

[0116] In one implementation, the first function value can be achieved in the following way:

[0117] Using the basic model, based on the input data of the first prediction model at the first time step, the first noise sample corresponding to the first time step is obtained, and the first noise sample is predicted through the remaining time steps to obtain the first prediction result; according to the consistency function, the first function value between the first noise sample and the first prediction result is obtained.

[0118] The first noise sample can be denoted as noise_nnet, and the first prediction result can be denoted as pred_x0_0.

[0119] Specifically, the consistency function can be expressed by formula (1):

[0120]

[0121] Where z is the noise we input to the model. It is denoising z, σ t and α t It is noiseschedule, which defines the noise variation strategy of the model during the noise addition and denoising process. skip and c out The boundary scaling factor is used to control z and Proportion.

[0122] Specifically, in this embodiment, the input data of the first prediction model at the first time step (i.e., the output data of the time step before the first time step, i.e., the denoising result of the first prediction model at the time step before the first time step) can be transmitted to the base model. In this way, the base model can obtain the first noise sample corresponding to the first time step through the assigned model parameters, and then continue to predict the remaining time steps (i.e., the time steps between the first time step and the last time step) from the first time step to obtain the noise sample obtained by the prediction of the last time step, i.e. the first prediction result. Finally, the consistency function of formula (1) is used to calculate the loss value between the first noise sample and the first prediction result, i.e. the first function value.

[0123] In one implementation, the second function value can be achieved in the following way:

[0124] First, using the teacher model, based on the input data of the first prediction model at the first time step, the second noise sample corresponding to the first time step is obtained, and the second noise sample is predicted through the remaining time steps to obtain the second prediction result; noise corresponding to multiple time steps is added to the second prediction result in sequence to obtain the noisy data corresponding to the first time step.

[0125] Then, using the student model, based on the noisy data corresponding to the first time step, a third noise sample corresponding to the second time step is obtained, and the third noise sample is predicted through the remaining time steps to obtain a third prediction result; according to the consistency function, a second function value between the third noise sample and the third prediction result is obtained.

[0126] The second noise sample can be denoted as noise_teacher, the second prediction result can be denoted as pred_x_0, and the noisy data can be denoted as x_prev.

[0127] Specifically, in this embodiment, the input data of the first prediction model at the first time step can be transmitted to the teacher model. In this way, the teacher model can obtain the second noise sample corresponding to the first time step through the assigned model parameters. Then, it can continue to predict the remaining time steps (the time steps between the first time step and the last time step) from the first time step to obtain the noise sample obtained by the prediction at the last time step, which is the second prediction result. Finally, the second prediction result is denoised multiple times and then denoised to the first time step to obtain the denoised data corresponding to the first time step.

[0128] The second noise sample includes noise samples with and without control elements. The control elements can be the input data of the first prediction model at the first time step out of T time steps, such as the input vector and relative position vector corresponding to the anchor view. Based on this, the second prediction result can be obtained by predicting the second noise sample over the remaining time steps, as follows:

[0129] The noise samples with and without control elements in the second noise sample are predicted using the remaining time steps to obtain the second prediction result.

[0130] It should be noted that the third noise sample can be denoted as noise_target. The third prediction result can be denoted as pred_x0_2.

[0131] Specifically, in this embodiment, the noise data corresponding to the first time step can be transmitted to the student model. In this way, the student model can obtain the third noise sample corresponding to the second time step through the assigned model parameters. Then, it can continue to predict the remaining time steps (i.e., the time steps between the second time step and the last time step) from the second time step to obtain the noise sample obtained by the prediction of the last time step, i.e. the third prediction result. Finally, the consistency function of formula (1) is used to calculate the loss value between the third noise sample and the third prediction result, i.e. the second function value.

[0132] Step 404: Obtain the second loss data based on the first function value and the second function value.

[0133] In this embodiment, a loss function can be used to calculate the loss between the first function value and the second function value to obtain the second loss data.

[0134] For example, the loss function can be a loss function based on the average bias error, or it can be the maximum likelihood loss function, and so on.

[0135] Step 405: Optimize the base model based on the second loss data to obtain the second prediction model.

[0136] Specifically, in this embodiment, the model parameters in the base model can be adjusted according to the second loss data. After multiple optimizations, the distilled potential consistency model, i.e., the second prediction model, can be obtained.

[0137] As can be seen, in this embodiment, the first prediction model can be distilled with input data at k time intervals to obtain the second prediction model. The second prediction model can reduce the prediction time steps, thereby speeding up the prediction rate and improving the efficiency of image generation when using the model.

[0138] In one implementation, after obtaining the second image in step 103, the second image can be attached to the three-dimensional unfolded view of the target object according to at least one calibrated position.

[0139] For example, taking a car as the target object, the first image is the main image of the car cover to be applied to the car cover, and the second image is the background image of the car cover extended from the main image of the car cover. After obtaining the background image of the car cover based on this embodiment, the car cover application area (i.e., the calibration position) on the three-dimensional unfolded image (i.e., UV image) of the car can be pre-calibrated, and the shape of the background image of the car cover is adjusted to the shape and size of the three-dimensional unfolded image. Based on this, the background image of the car cover is applied according to the corresponding application area of ​​the car cover. At this time, the three-dimensional unfolded image is displayed as pure white in the mask image. Thus, the white part of the mask in the three-dimensional unfolded image is replaced with the background image of the car cover. This embodiment can thus be implemented.

[0140] The above describes an image generation method provided by an embodiment of this application. The following describes the apparatus for performing the above image generation method.

[0141] Please see Figure 5 , Figure 5 This is a schematic diagram of an image generation apparatus provided in an embodiment of this application. The apparatus can be configured in an electronic device capable of data processing, such as a computer or server. The technical solution in this embodiment is mainly used to achieve shape controllability in image generation.

[0142] Specifically, the device in this embodiment may include the following units:

[0143] Vector encoding unit 501 is used to encode the first image and the first shape information to obtain a first vector;

[0144] The denoising prediction unit 502 is configured to transmit at least the noisy image and the first vector to the target prediction model to obtain the denoising prediction result obtained by the target prediction model based on the first shape information and the second shape information; the second shape information may be the same as or different from the first shape information.

[0145] Image decoding unit 503 is used to decode the denoising prediction result to obtain a second image, the second image having the second shape information.

[0146] According to the above technical solution, the image generation apparatus provided in this application first encodes the first image and the first shape information. This allows the obtained first vector and the noisy image to be transmitted to the target prediction model. The target prediction model then obtains a denoising prediction result based on second shape information that is the same as or different from the first shape information. After decoding, the second image corresponding to the second shape information is obtained. Therefore, in this embodiment, the first shape information of the first image is included in the image encoding process. This allows the target prediction model to output a corresponding denoising prediction result based on the first shape information and the second shape information input into the model. After decoding, the image corresponding to the second shape information is obtained, thus achieving shape-controllable image generation.

[0147] In one implementation, the target prediction model is a first prediction model or a second prediction model; the first prediction model obtains the denoised prediction result after T time steps; T is a positive integer; the second prediction model obtains the denoised prediction result after S time steps; S is a positive integer; the second prediction model is a latent consistency model obtained by distilling the first prediction model; wherein S is less than T.

[0148] In one implementation, this embodiment may further include the following units, such as... Figure 6 As shown:

[0149] Model training unit 504 is used to train the first prediction model in the following manner:

[0150] In a sample image, an anchor view and a target view are extracted; the anchor view has input shape information, and the target view has output shape information; the anchor view is encoded according to the input shape information to obtain an input vector; the target view with the added noise image is encoded according to the output shape information to obtain an output vector; at least the noise image and the input vector are transmitted to the first prediction model to obtain a sample prediction result obtained by the first prediction model after T time steps; a first loss data is obtained based on the sample prediction result and the output vector; the first prediction model is optimized based on the first loss data.

[0151] In one implementation, this embodiment may further include the following units, such as... Figure 7 As shown:

[0152] The model distillation unit 505 is used to distill the first prediction model to obtain a second prediction model in the following manner: The model parameters of the first prediction model are imported into a pre-constructed base model, teacher model, and student model; the base model, teacher model, and student model are all neural network models; In T time steps of the first prediction model, multiple first time steps and a second time step corresponding to each first time step are sampled, the second time step being after the first time step and spaced k time steps apart from the first time step; k is a positive integer; For each first time step, the following processing is performed: The input data of the first prediction model at the first time step is processed using the base model, teacher model, and student model to obtain a first function value corresponding to the first time step and a second function value corresponding to the second time step; Second loss data is obtained based on the first function value and the second function value; The base model is optimized based on the second loss data to obtain the second prediction model.

[0153] In one implementation, the second noise sample includes: a noise sample with added control elements and a noise sample without added control elements; the control element is: the input data of the first prediction model at the first time step in the T time steps;

[0154] Specifically, when the model distillation unit 505 performs a remaining time step prediction on the second noise sample to obtain a second prediction result, it is used to perform a remaining time step prediction on the noise sample with and without control elements in the second noise sample to obtain a second prediction result.

[0155] In one implementation, this embodiment may further include the following units, such as... Figure 8 As shown:

[0156] The image bonding unit 506 is used to bond the second image to the three-dimensional unfolded image of the target object according to at least one calibrated position after the image decoding unit 503 obtains the second image.

[0157] It should be noted that the specific implementation of each unit in this embodiment can be referred to the corresponding content above, and will not be described in detail here.

[0158] This application also provides an electronic device in its embodiments. (See reference...) Figure 9 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 9The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0159] like Figure 9 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. When the electronic device is powered on, the RAM 903 also stores various computer programs and data required for the operation of the electronic device. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904. In this embodiment, the processing unit 901, i.e., the processor, is used to execute computer programs so that the electronic device can implement any of the image generation methods provided in the embodiments of this application.

[0160] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, memory cards, hard drives, etc.; and communication devices 909. Communication device 909 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0161] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the image generation methods provided in this application.

[0162] This application also provides a computer storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the image generation methods provided in this application.

[0163] The following example illustrates the technical solution of this application: The scenario where the main image of a car cover is expanded into a background image is used as an example.

[0164] This application proposes a diffusion model-based image expansion scheme that combines a denoising method based on a latent consistency model with PQDiff. This scheme cleverly utilizes the latent consistency model to directly predict solutions in the latent space, reducing the need for iterative denoising and significantly improving the speed of image generation. Furthermore, this application incorporates the expected pixel size information of the generated image (i.e., the shape information mentioned earlier) into the UViT Encoder layer, allowing control over the pixel shape of the generated image and thus improving its quality.

[0165] The technical solution proposed in this application aims to generate high-quality epitaxial rendering images through a diffusion model, and to control their shape and accelerate image generation, so as to solve the problems of fixed image shape, slow speed and semantic deviation in the prior art.

[0166] Figure 10 Technology roadmap for UViT training:

[0167] First, the basic scheme of this application is divided into two main stages:

[0168] 1. Training Phase:

[0169] During the training phase, this application incorporates the shapes of the target view and anchor view (i.e., input shape information and output shape information) into the Encoder layer. When transforming the image into a latent vector, the shape information is included. Then, leveraging the shape-insensitive nature of the UViT diffusion model, the shape information is preserved and finally released in the Decoder layer. Simultaneously, the PatchEmbed layer in UViT also divides the image vector into blocks based on the shape information.

[0170] Specifically, such as Figure 10 As shown, this application randomly extracts anchor and target views from the input image. The relative positional embedding algorithm is used to calculate the relationship between the anchor and target views, i.e., their relative position vectors. Then, both the anchor and target views are simultaneously fed into the encoder layer. This approach reduces computation by placing the image into an implicit space and converting it into vectors. The two image vectors and the relative position vector between the images are then fed into UViT. Noise is added to the target view vector as real noise. By training UViT, this application aims to train a UViT that can predict the noise.

[0171] Subsequently, Latent Consistency Distillation (LCD) was performed using the pre-trained UViT. The images were then converted to latent space using the same encoder, and the skip interval k was set to 20 based on previous research.

[0172] For each image vector, random sampling is performed at time steps [1, Nk] (where N is the total number of time steps, as T mentioned earlier). The number of samples is equal to the batch size, resulting in the sequence of the starting time steps, denoted as start_timesteps. Each item in the starting time steps is subtracted by k to obtain the second time step, denoted as timesteps. Then, the boundary scaling factor c for start_timesteps and timesteps is obtained respectively.

[0173] Next, the trained UViT model is imported into three neural network models: nnet, teacher_nnet, and target_nnet. Teacher_nnet and target_nnet do not participate in backpropagation of gradients. In this embodiment, nnet can first use the previously trained UViT to predict noise, and then gradually denoise the noise to obtain a predicted original image.

[0174] Using formula (1), the consistency equation obtained by nnet is derived, and the result is denoted as follows: Similarly, teacher_nnet is used to predict noise with and without control elements (anchor view vector and relative position relationship vector), and noise is removed separately. Then, CFG scale w (cue word relevance, which refers to the degree of influence of cue words on image generation) is introduced, and the formula is used... Where t is timesteps, w represents unconditional noise, and c is the guidance scale. Indicates no control vector, ∈ θ (z t (c, t) represents conditional noise. This leads to the introduction of w and The predicted noise, `pred_noise`, is the noise predicted by `teacher_nnet` at the `start_timesteps` time step. Similarly, replacing the noise prediction model ∈ with x0, i.e., `pred_x0 = (1 + w)x0 - wx0`, we can obtain the result by introducing w and... The original image predicted is pred_x0. Then, noise is added using pred_noise, pred_x0, and time steps to obtain x_prev. In other words, noise is added to the image vector pred_x0 after noise denoising using teacher_nnet prediction, and the noise is added up to time step t_n+k to obtain x_prev.

[0175] Finally, x_prev is used as input to target_nnet to predict target_noise_predict (i.e., the noise at timesteps predicted by target_nnet). Here, the timestep sequence used later is timesteps, while the previous steps all use start_timesteps. Then, target_noise_predict is denoised to obtain pred_x_0. That is, pred_noise is used to calculate the noise x_prev at start_timesteps, then x_prev is used as input to target to predict the noise at timesteps, and finally denoised to obtain pred_x_0. Finally, c_skip*x_prev + c_out*pred_x_0 is used to obtain... Finally, the loss function is calculated, which is the same as the consistency function calculated by NNet. Then, gradient backpropagation is performed for optimization, and upon convergence, a latent consistency model suitable for image expansion is obtained.

[0176] 2. Combine generated images with 3D car models:

[0177] Various segmentation models can be selected to extract the main image of the car cover from the provided car cover material image. By using the main car cover image as input to the model mentioned earlier, such as the latent consistency model, the trained model can be used to expand the image and generate the background image of the car cover required for the car model. The car cover bonding area on the UV unwrapped map is calibrated so that it appears as pure white in the mask image, while the shape of the generated car cover background image is adjusted to the size of the UV unwrapped map. Then, the car cover background image and the car cover bonding area are combined, and the white parts in the mask are replaced with the corresponding parts of the car cover background image.

[0178] By adopting the above two stages, not only is the speed and quality of map expansion generation improved, but semantic bias caused by excessive map expansion is also avoided, making it suitable for various application scenarios and putting the model into practice for generating 3D car model car covers.

[0179] It is evident that this application has the following technical advantages:

[0180] (1) High-quality expanded image generation: This application can generate expanded images with harmonious content by utilizing the establishment of relative position embedding vectors. This means that the content and quality of the expanded image can accurately reflect the user's ideas and meet the application scenarios with high precision requirements, such as user-defined associated image generation, car model and car cover generation, etc.

[0181] (2) Rich Detail Representation: This application is based on a diffusion model, trained with a large amount of data and reasonable parameter settings, which enables the model to learn rich image features and texture information. During sampling, noise is gradually removed, and specific structures and details of the data are gradually introduced from the noise. Therefore, the generated results have rich detail representation.

[0182] (3) High efficiency in map expansion generation: This application solves the problem of slow sampling speed of diffusion models in map expansion. By introducing latent consistency distillation and latent consistency sampling methods, the generation speed is greatly improved.

[0183] (4) Wide application adaptability: The technical solution of this application is not only applicable to the traditional field of image extension, but can also be extended to multiple fields such as virtual reality, augmented reality, game development, and film production, providing users with a richer and more immersive experience.

[0184] In summary, the technical effects of this application not only represent a significant breakthrough in technology, but also demonstrate remarkable advantages in practical applications, providing users with a higher-quality image extension drawing solution.

[0185] In summary, this application aims to solve a series of technical problems existing in current image extension rendering techniques, particularly the challenges in image generation speed and the shape and quality of generated images. Furthermore, it aims to practically apply the extended image content as a background for a car wrap. Specifically, the technical solutions and objectives can be summarized as follows:

[0186] 1) Controllable Generation of Image Shapes: Achieving the desired image expansion result using the same pre-trained model has always been a challenge. This application achieves controllable generation of image shapes by adding shape information to the Encoder and Decoder layers and adjusting the Patch_Embed layer in the UViT layer to correspond to the shape information.

[0187] 2) Avoiding semantic bias in image expansion: To address the semantic bias problem commonly encountered in current image expansion models at high magnification, this application solves the semantic problem by calculating the relative position embedding between the anchor view and the target view. By embedding vectors based on relative position, the generated content becomes more harmonious.

[0188] 3) Significantly improve the speed of map expansion generation: By distilling the consistency function required for LCM from the pre-trained modified PQDiff model (this process can be called LCD), this application can directly predict the solution of the latent space, which greatly reduces the number of iterations for denoising and improves the generation speed.

[0189] 4) Combine the generated image with the 3D car model: By providing the main image of the car cover to the model, expand the image to generate the background image of the entire car cover, and then combine it with the effective parts of the car's UV unfolded image to generate the 3D car cover for the car model.

[0190] In summary, the technical solution of this application aims to solve key problems in existing image extension rendering techniques by introducing the shape information of the expected generated image and a multi-step potential consistency sampling algorithm. These problems include controllable image shape generation, avoiding semantic deviations caused by image expansion, improving the speed of image expansion generation, and realistically combining the expanded image with the 3D car model. The implementation of these technologies will promote the development of the field of image extension rendering and provide the automotive industry with a higher quality and more efficient 3D car model and car cover generation solution.

[0191] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0192] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0193] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0194] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An image generation method, characterized in that, The method includes: The first image and the first shape information are encoded to obtain the first vector; At least the noisy image and the first vector are transmitted to the target prediction model to obtain the denoising prediction result obtained by the target prediction model based on the first shape information and the second shape information; the second shape information may be the same as or different from the first shape information; The denoising prediction result is decoded to obtain a second image, which has the second shape information.

2. The method according to claim 1, characterized in that, The target prediction model is either a first prediction model or a second prediction model; The first prediction model obtains the denoised prediction result after T time steps; T is a positive integer; the second prediction model obtains the denoised prediction result after S time steps; S is a positive integer; the second prediction model is a latent consistency model obtained by distilling the first prediction model; Where S is less than T.

3. The method according to claim 2, characterized in that, The first prediction model was trained in the following way: In the sample image, an anchor view and a target view are cropped; the anchor view has input shape information, and the target view has output shape information; The anchor view is encoded based on the input shape information to obtain an input vector; Based on the output shape information, the target view with the added noise image is encoded to obtain an output vector; The noisy image and the input vector are transmitted to the first prediction model at least to obtain the sample prediction result obtained by the first prediction model after T time steps; Based on the sample prediction results and the output vector, the first loss data is obtained; Based on the first loss data, the first prediction model is optimized.

4. The method according to claim 2 or 3, characterized in that, The second prediction model was obtained in the following way: The model parameters of the first prediction model are imported into a pre-built base model, teacher model, and student model; the base model, teacher model, and student model are all neural network models. In the T time steps of the first prediction model, multiple first time steps and a second time step corresponding to each first time step are sampled. The second time step is after the first time step and is separated from the first time step by k time steps; k is a positive integer. For each of the first time steps, perform the following processing: Using the base model, the teacher model, and the student model, the input data of the first prediction model at the first time step is processed to obtain the first function value corresponding to the first time step and the second function value corresponding to the second time step; Based on the first function value and the second function value, the second loss data is obtained; Based on the second loss data, the base model is optimized to obtain the second prediction model.

5. The method according to claim 4, characterized in that, The second noise sample includes: noise samples with added control elements and noise samples without added control elements; the control element is: the input data of the first prediction model at the first time step in the T time steps; The second noise sample is predicted using the remaining time step to obtain a second prediction result, including: The noise samples with and without control elements in the second noise sample are predicted using the remaining time steps to obtain the second prediction result.

6. The method according to claim 1 or 2, characterized in that, After decoding the denoised prediction result to obtain the second image, the method further includes: The second image is overlaid onto the three-dimensional unfolded image of the target object at at least one calibrated position.

7. The method according to claim 1 or 2, characterized in that, At least the noisy image and the first vector are transmitted to the target prediction model to obtain the denoising prediction result obtained by the target prediction model based on the second shape information, including: The noisy image, the first vector, and the relative position vector are transmitted to the target prediction model to obtain the denoising prediction result obtained by the target prediction model based on the second shape information and the relative position vector; The second image and the first image are in a relative positional relationship corresponding to the relative position vector.

8. An image generation apparatus, characterized in that, include: A vector encoding unit is used to encode the first image and the first shape information to obtain a first vector; A denoising prediction unit is configured to transmit at least the noisy image and the first vector to the target prediction model to obtain a denoising prediction result obtained by the target prediction model based on the first shape information and the second shape information. The second shape information may be the same as or different from the first shape information; An image decoding unit is used to decode the denoising prediction result to obtain a second image, the second image having the second shape information.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the image generation method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the image generation method as described in any one of claims 1 to 6.

Images