View angle controllable image generation method based on 3D information injection

Abstract

The invention discloses a visual angle controllable image generation method based on 3D information injection, and aims at overcoming the defects of an existing image generation model in the aspect of specific object visual angle control. By fusing 3D information and text description, the method can generate a vivid specific object image under diversified scenes and backgrounds. According to the method, models such as a multi-layer perceptron and a transformer are adopted to process visual angle information and text features, and then condition-controlled image generation is achieved through U-Net of a table diffusion model. In addition, a pose prediction module and a new loss function are introduced, so that the visual angle accuracy is improved, and the adaptability of the model and the individuation degree of the generated image are enhanced. The implementation of the method is expected to promote the development of the image generation technology in the aspects of personalization and controllability.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image generation, and in particular relates to a method for generating an image with controllable viewing angle based on 3D information injection. Background Art

[0002] With the remarkable achievements of diffusion models in the field of image generation, generative models are now able to create high-quality and diverse images. Based on this, diffusion models for large-scale text-to-image generation have shown excellent capabilities, which can generate images based on natural language prompts. However, these models still have shortcomings in understanding specific objects and generating images in different contexts.

[0003] In order to improve the practicality and personalization of generative models, fine-tuning methods for models have emerged one after another. The purpose of these fine-tuning methods is to make the model learn specific objects. These methods use a small number of input images to achieve personalized training of text-to-image diffusion models, mainly by adding new identifiers to the pre-trained model and fine-tuning it to generate realistic images of specific objects in different scenes and backgrounds. These methods take advantage of the powerful semantic prior capabilities of the diffusion model to learn specific objects and accurately generate them in a variety of scenes, postures, and lighting conditions.

[0004] However, the downside is that although these methods can generate images of specific objects, their ability to control the viewpoint of the object is still limited. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method for generating images with controllable viewing angle based on 3D information injection, which solves the problem of the lack of object viewing angle control when generating images of specific objects in the prior art.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] The method for generating a view-controllable image based on 3D information injection comprises the following steps:

[0008] Step S1, obtaining a training set consisting of a batch of input triplets according to a 3D model of a specific object, each triplet including a rendering, viewing angle information, and a text description;

[0009] Step S2: Pass the rotation matrix describing the object's perspective through a multi-layer perceptron, and replace the placeholders in the text with the result of the multi-layer perceptron, and then obtain the text features through a transformer;

[0010] Step S3, input the obtained text features into the U-Net in stable diffusion as control conditions to obtain a controlled generated image;

[0011] Step S4, applying a pre-trained pose prediction module to predict the viewing angle of a specific object in the generated image;

[0012] Step S5: Calculate the loss function based on the generated image, predicted viewing angle information, input image, and real viewing angle, and set training parameters to train the models in S1-S4.

[0013] In step S1, a public data set with 3D models is selected, and images at different perspectives are obtained through random sampling. The perspective of the object is recorded using a rotation matrix. After the rendering is obtained, a text description of each rendering is obtained according to the image-to-text model.

[0014] Expand the text description. The specific format is:

[0015] “a[placeholder][identifier][class noun][background]”; where [placeholder], [identifier], [class noun], and [background] are a placeholder, a special identifier, a coarse-grained class description of a specific object, and background information, respectively.

[0016] In step S2, each text description is passed through a tokenizer to obtain a batch of word blocks. The embedding vector of each word block can be obtained by querying. The result after the rotation matrix passes through the multi-layer perceptron is used as the embedding vector of the placeholder, and then the features of the entire text are obtained through a transformer.

[0017] The specific design method of the multi-layer perceptron is as follows:

[0018] First, the rotation matrix is ​​flattened into a 9-dimensional one-dimensional vector through a flattening layer;

[0019] Then, the obtained one-dimensional vector is passed through two fully connected linear layers in sequence to map it to 512-dimensional and 512-dimensional spaces respectively, and each linear layer is followed by a ReLU activation function;

[0020] Finally, an output linear layer is used to map the 512-dimensional representation to the target dimension and output it.

[0021] The specific process of step S3 is as follows:

[0022] The text features are injected into the cross-attention network of U-Net. The cross-attention network is constructed by inputting the text feature vector and the intermediate representation of the image in U-Net Processing to effectively integrate text information and image content;

[0023] The attention network calculates vectors K and V based on the text feature c, calculates vector Q based on the intermediate representation z of the image, and calculates the cross attention mechanism through the following formula to finally obtain the image features after interaction:

[0024]

[0025] Where Q = W q z,K=W k c, V = W v c; is the weight matrix; d ′ is the dimension of the Q, K, V outputs.

[0026] In step S4, the pose prediction module has been trained. The parameters of this module are frozen during training and do not participate in the back-propagation parameter update. This module only needs to input the generated image and obtain the pose of the target in the input image based on the reference image set and the corresponding pose. The reference image set is from all the images in the training set. The pose prediction module consists of three parts, namely, the object detector, the viewpoint selector, and the pose refiner. The object detector is used to detect the local area of ​​the object and estimate the translation depth of the object by the area size.

[0027] The viewpoint selector is used to get a rough rotation perspective by selecting the reference image that is most similar to the input image;

[0028] The pose refiner is used to update the coarse estimated translation and rotation to the precise translation and rotation.

[0029] The loss function in step S5 includes a noise prediction square error loss term of a stable diffusion model and a perspective preservation loss term, and the loss function is calculated by the following formula:

[0030]

[0031] in, is the noise prediction squared error loss term, The view preservation loss term, λ controls The relative weight of

[0032] Based on the actual noise and the predicted value of the noise, the squared error loss term is calculated by the following formula:

[0033]

[0034] Among them, z t is the hidden code at time step t, c(·) is the condition used to process the input, representing the rotation matrix of the view and the text description;

[0035] The view preservation loss is calculated based on the true rotation matrix corresponding to the object in the image and the predicted pose of the object using the following formula:

[0036]

[0037] Among them, P gt refers to the true viewing angle of the input image, refers to the pose prediction module, x ′ Represents an image generated by the diffusion model.

[0038] The training parameters in step S5 include a learning rate and training rounds. The models in S1-S4 are iteratively trained by setting the training parameters until a maximum number of training rounds is reached.

[0039] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, call all or part of the steps of the method.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. The present invention inserts the perspective information into the text features through transformation and adds it to the diffusion model as a control condition, thereby realizing the control of the object perspective, making the generated results more personalized, and also increasing the controllability of image generation.

[0042] 2. The present invention proposes a new loss term for maintaining the target perspective. It obtains the pose of the target in the generated image through a pose prediction module, and compares the perspective information (rotation matrix) in the result with the true value, thereby enhancing the model's perception of perspective and making the generated object perspective more accurate.

[0043] 3. The present invention combines perspective information and text features and introduces a new loss term for perspective, which not only improves the perspective accuracy of the generated image, but also enhances the adaptability of the model to complex scenes and diverse environments, so that the model can generate images of specific objects in different perspectives and different scenes based on its own powerful generation ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of the training set triples of the present invention.

[0045] Figure 2 The figure is a schematic diagram of the workflow of the method for generating an image with controllable viewing angle based on 3D information injection according to the present invention.

[0046] Figure 3 This is a structural diagram of the network model during training of the present invention. DETAILED DESCRIPTION

[0047] The structure and working process of the present invention will be further described below in conjunction with the accompanying drawings.

[0048] In view of the limitations and shortcomings of the existing technology, the present invention proposes a method for generating a controllable perspective image using 3D information and text description. First, the 3D information and text description are fused, and features are extracted through a multi-layer perceptron and transformer. The features are then injected into stable diffusion to generate realistic, controllable perspective object images in different scenes. This method improves the perspective accuracy and image personalization, and provides a new perspective control method for the field of image generation.

[0049] The method for generating a view-controllable image based on 3D information injection comprises the following steps:

[0050] Step S1, obtaining a training set consisting of a batch of input triplets according to a 3D model of a specific object, each triplet including a rendering, viewing angle information, and a text description;

[0051] Step S2: Pass the rotation matrix describing the object's perspective through a multi-layer perceptron, and replace the placeholders in the text with the result of the multi-layer perceptron, and then obtain the text features through a transformer;

[0052] Step S3, input the obtained text features into the U-Net in stable diffusion as control conditions to obtain a controlled generated image;

[0053] Step S4, applying a pre-trained pose prediction module to predict the viewing angle of a specific object in the generated image;

[0054] Step S5: Calculate the loss function based on the generated image, predicted viewing angle information, input image, and real viewing angle, and set training parameters to train the models in S1-S4.

[0055] Specific embodiments, such as Figures 1 to 3 As shown,

[0056] The method for generating a view-controllable image based on 3D information injection has the following specific steps:

[0057] S1: Based on the 3D model of a specific object, a training set consisting of a batch of input triplets is obtained, each of which includes a rendering, view information, and a text description. The text description is then expanded according to a specific format;

[0058] We select a public dataset with 3D models and obtain a training set consisting of a batch of input triplets based on the 3D model of a specific object. Each triplet includes a rendering, view information, and a text description. Examples of triples are as follows: Figure 1 As shown in Figure 1. A batch of renderings at different perspectives can be obtained through random sampling. The perspective of the object in the image is represented by a rotation matrix, and then the text description of each rendering is obtained based on the image-to-text model. In this way, a rendering, a rotation matrix, and a text description form an input triple. A batch of different input triplets can be obtained through random sampling, and this batch of triplets constitutes the training set.

[0059] The specific format of the expanded text description is: "a[placeholder][identifier][class noun][background]". Among them, "[placeholder]", "[identifier]", "[class noun]", and "[background]" are placeholders, special identifiers, coarse-grained category description words for specific objects, and background information, respectively. The special identifier is a rare identifier queried from the vocabulary and then converted back to the text space. It is worth noting that the identifier should avoid using common English words because they have strong semantic priors and need to be decoupled from their original meanings and reassociated to specific objects; at the same time, it should also avoid using random combinations of English letters to generate identifiers, because even if they are randomly combined English letters, the model's tokenizer may separate them separately, and these letters may have strong prior knowledge for the diffusion model.

[0060] S2: Pass the rotation matrix describing the object’s perspective through a multi-layer perceptron, and replace the placeholders in the text with the result of the multi-layer perceptron. Then, pass it through a transformer to get the text features.

[0061] The text description is passed through a tokenizer to obtain a batch of tokens. The embedding vector of each token can be obtained by querying. At this time, the placeholder does not actually correspond to an embedding vector and needs to be supplemented according to the perspective information. The rotation matrix describing the perspective is passed through a multi-layer perceptron, and the result is used as the embedding vector of the placeholder to supplement the completeness of the text. Then, the features of the entire text are obtained through a transformer.

[0062] The specific structure of the multilayer perceptron is as follows, but it is not limited to this structure: First, a Flatten layer is used to flatten the matrix into a 9-dimensional one-dimensional vector. Next, this vector passes through two fully connected linear layers in turn, mapping it to 512-dimensional and 512-dimensional spaces respectively. Each linear layer is followed by a ReLU activation function to increase nonlinearity. Finally, an output linear layer maps the 512-dimensional representation to the target dimension and outputs it.

[0063] S3: Input the obtained text features into the U-Net in stable diffusion as control conditions to obtain the controlled generated images;

[0064] The obtained text features are input as conditions into the cross attention network of U-Net in the stable diffusion model. The cross attention network is and the intermediate representation of the image in U-Net The attention network calculates vectors K and V based on the text feature c, calculates vector Q based on the intermediate representation z of the image, and calculates the cross-attention mechanism through the following formula to finally obtain the image features after interaction:

[0065]

[0066] In the formula 1, Q = W q z,K=W k c, V = W v c; is the weight matrix; d ′ is the dimension of the Q, K, V outputs.

[0067] After a certain denoising process using U-Net, the final image is obtained.

[0068] S4: A pose prediction module predicts the viewpoint of a specific object in the generated image;

[0069] Based on a pose prediction module, the viewpoint of a specific object in the generated image is predicted. This module only needs to input the generated image and obtain the pose of the target in the input image based on the reference image set and the corresponding pose. The reference image set is all the images in the training set. This module is built on the Gen6D model, but is not limited to this model. The pose prediction module based on the Gen6D model consists of three parts: detector, selector and refiner. Their functions are as follows: detect the local area of ​​the object and estimate the depth (translation) of the object by the size of the area; obtain a rough viewpoint (rotation) by selecting the reference image that is most similar to the query image; update the roughly estimated translation and rotation to precise translation and rotation.

[0070] It is worth noting that the pose prediction module has been trained, and the parameters of this module are frozen during training and do not participate in back propagation parameter updates. This module only needs to input an RGB image and obtain the pose of the target in the input image based on the reference image set and the corresponding pose. The reference image set is from all images in the training set.

[0071] S5: Calculate the loss function based on the generated image, predicted view information, input image, and real view, and set training parameters to train the models in S1-S4;

[0072] The generated image and the predicted perspective information are compared with the true value and the loss is calculated. The loss function includes a stable diffusion noise prediction square error loss term and a perspective preservation loss term. The loss function is calculated by formula 2, which is expressed as:

[0073]

[0074] In the formula 2, is the noise prediction squared error loss term, The view preservation loss term, λ controls relative weight of .

[0075] Based on the actual noise and the predicted value of the noise, the square error loss term can be calculated by Formula 3, which is expressed as:

[0076]

[0077] In the above formula 3, z t is the latent code at time step t, and c(·) is the condition used to process the input (rotation matrix representing the viewpoint and text description).

[0078] The view preservation loss is calculated based on the true rotation matrix corresponding to the object in the image and the predicted pose of the object by formula 4, which is expressed as:

[0079]

[0080] In the above formula, P gt refers to the true viewing angle of the input image, refers to the pose prediction module, x ′ Represents an image generated by the diffusion model.

[0081] Set reasonable training parameters, perform gradient back propagation according to the loss minimization principle, and update network parameters. The specific structure of the entire network is as follows: Figure 3 As shown, a complete neural network training process is realized.

[0082] Furthermore, in S5, the training parameters are learning rate and training rounds, and S5 specifically includes: iteratively training the models in S1-S4 by setting the training parameters until a maximum number of training rounds is reached.

[0083] This method aims to address the shortcomings of existing image generation models in controlling the viewpoint of specific objects. By fusing 3D information and text descriptions, this method is able to generate realistic images of specific objects in a variety of scenes and backgrounds. It uses models such as multi-layer perceptrons and transformers to process viewpoint information and text features, and then achieves conditionally controlled image generation through the U-Net of the stable diffusion model. In addition, the introduction of a pose prediction module and a new loss function not only improves the viewpoint accuracy, but also enhances the adaptability of the model and the personalization of the generated images. The implementation of this method is expected to promote the development of image generation technology in terms of personalization and controllability.

[0084] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0085] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A method for generating a view-controlled image based on 3D information injection, characterized in that: The steps include: Step S1, obtaining a training set consisting of a batch of input triplets according to a 3D model of a specific object, each triplet including a rendering, viewing angle information, and a text description; Step S2: Pass the rotation matrix describing the object's perspective through a multi-layer perceptron, and replace the placeholders in the text with the result of the multi-layer perceptron, and then obtain the text features through a transformer; Step S3, inputting the obtained text features into the U-Net in the stable diffusion model as control conditions to obtain a controlled generated image; Step S4, applying a pre-trained pose prediction module to predict the viewing angle of a specific object in the generated image; Step S5: Calculate the loss function based on the generated image, predicted viewing angle information, input image, and real viewing angle, and set training parameters to train the models in S1-S4.

2. The method for generating an image with controllable viewing angle based on 3D information injection according to claim 1, characterized in that: In step S1, a public data set with 3D models is selected, and images at different perspectives are obtained through random sampling. The perspective of the object is recorded using a rotation matrix. After the rendering is obtained, a text description of each rendering is obtained according to the image-to-text model.

3. The method for generating an image with controllable viewing angle based on 3D information injection according to claim 2, characterized in that: Expand the text description. The specific format is: "a[placeholder][identifier][class noun][background]"; where [placeholder], [identifier], [class noun], and [background] are placeholder, special identifier, coarse-grained class description for a specific object, and background information, respectively.

4. The method for generating an image with controllable viewing angle based on 3D information injection according to claim 1, characterized in that: In step S2, each text description is passed through a tokenizer to obtain a batch of word blocks. The embedding vector of each word block can be obtained by querying. The result after the rotation matrix passes through the multi-layer perceptron is used as the embedding vector of the placeholder, and then the features of the entire text are obtained through a transformer.

5. The method for generating an image with controllable viewing angle based on 3D information injection according to claim 4, characterized in that: The specific design method of the multi-layer perceptron is as follows: First, the rotation matrix is ​​flattened into a 9-dimensional one-dimensional vector through a flattening layer; Then, the obtained one-dimensional vector is passed through two fully connected linear layers in sequence to map it to 512-dimensional and 512-dimensional spaces respectively, and each linear layer is followed by a ReLU activation function; Finally, an output linear layer is used to map the 512-dimensional representation to the target dimension and output it.

6. The method for generating an image with controllable viewing angle based on 3D information injection according to claim 1, characterized in that: The specific process of step S3 is as follows: The text features are injected into the cross-attention network of U-Net. The cross-attention network is constructed by inputting the text feature vector and the intermediate representation of the image in U-Net Processing to effectively integrate text information and image content; The attention network calculates vectors K and V based on the text feature c, calculates vector Q based on the intermediate representation z of the image, and calculates the cross attention mechanism through the following formula to finally obtain the image features after interaction: Where Q = W q z,K=W k c, V = W v c; is the weight matrix; d′ is the dimension of Q, K, V outputs.

7. The method for generating an image with controllable viewing angle based on 3D information injection according to claim 1, characterized in that: In step S4, the pose prediction module includes three parts, namely, object detector, viewpoint selector, and pose refiner, where: The object detector is used to detect the local area of ​​the object and estimate the translation depth of the object by the size of the area; The viewpoint selector is used to get a rough rotation perspective by selecting the reference image that is most similar to the input image; The pose refiner is used to update the coarse estimated translation and rotation to the precise translation and rotation.

8. The method for generating an image with controllable viewing angle based on 3D information injection according to claim 1, characterized in that: The loss function in step S5 includes a noise prediction square error loss term of a stable diffusion model and a perspective preservation loss term, and the loss function is calculated by the following formula: in, is the noise prediction squared error loss term, The perspective preservation loss term, λ controls The relative weight of Based on the actual noise and the predicted value of the noise, the squared error loss term is calculated by the following formula: Among them, z t is the hidden code at time step t, c(·) is the condition used to process the input, which includes the rotation matrix representing the viewpoint and the text description; The view preservation loss is calculated based on the true rotation matrix corresponding to the object in the image and the predicted pose of the object using the following formula: Among them, P gt refers to the true viewing angle of the input image, refers to the pose prediction module, and x′ represents the image generated by the diffusion model.

9. The method for generating an image with controllable viewing angle based on 3D information injection according to claim 1, characterized in that: The training parameters in step S5 include a learning rate and training rounds. The models in S1-S4 are iteratively trained by setting the training parameters until a maximum number of training rounds is reached.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, all or part of the steps of the method described in any one of claims 1 to 9 are called.

Images