Image processing method and device

Abstract

The invention discloses an image processing method and device, and the method comprises the steps: obtaining description information of a to-be-generated image and a first image feature; the description information comprises visual angle information corresponding to the to-be-generated image; the first image feature represents an image feature containing noise information; determining a target view angle control vector based on the view angle information; based on the description information and the target visual angle control vector, performing denoising processing on the first image feature to obtain a target image feature; and generating a target image based on the target image features.

Description

Technical Field

[0001] The present application relates to, but is not limited to, the field of computer technology, and in particular to an image processing method and device. Background Art

[0002] With the development of deep learning technology, text-generated image technology has been widely used in many fields. However, current technology obviously lacks accuracy and consistency when processing multi-angle view generation, especially in understanding and expressing complex three-dimensional scenes, and cannot even generate images consistent with the perspective in the text description. Summary of the invention

[0003] In view of this, the present application at least provides an image processing method and device.

[0004] The technical solution of this application is implemented as follows:

[0005] In one aspect, the present application provides an image processing method, the method comprising:

[0006] Acquire description information and a first image feature of the image to be generated; the description information includes viewing angle information corresponding to the image to be generated; the first image feature represents an image feature containing noise information;

[0007] Based on the perspective information, determine the target perspective control vector;

[0008] Based on the description information and the target viewing angle control vector, a denoising process is performed on the first image feature to obtain a target image feature;

[0009] Based on the target image features, a target image is generated.

[0010] In some embodiments, determining a target perspective control vector based on the perspective information includes:

[0011] Determine a first viewing angle control vector corresponding to the viewing angle information; the first viewing angle control vector represents a three-dimensional vector used to control the image generation angle;

[0012] The first viewing angle control vector is projected into a two-dimensional second viewing angle control vector, and the second viewing angle control vector is used as a target viewing angle control vector.

[0013] In some embodiments:

[0014] Determining a target perspective control vector based on the perspective information includes: determining the target perspective control vector based on the perspective information using a perspective control network in a Wensheng graph model;

[0015] Based on the description information and the target perspective control vector, performing denoising processing on the first image feature to obtain the target image feature, including: based on the description information and the target perspective control vector, using a denoising network in a Vincent graph model, performing denoising processing on the first image feature to obtain the target image feature;

[0016] Based on the target image features, a target image is generated, including: based on the target image features, a decoder network in a text-based graph model is used to generate the target image.

[0017] In some embodiments, the image processing method further includes:

[0018] For each sample description information in the sample set, a teacher model is used to generate a sample image feature corresponding to the sample description information; each sample description information includes corresponding sample viewing angle information; the sample image feature represents a two-dimensional image feature corresponding to the sample description information under the sample viewing angle;

[0019] For each sample description information, a student model is used to generate a predicted image feature corresponding to the sample description information; the student model includes a first neural network and a denoising network; the first neural network is used to generate a predicted perspective control vector based on the sample perspective information;

[0020] Determine the loss information between the predicted image features and the sample image features corresponding to each sample description information;

[0021] Based on the loss information, the first neural network is updated to obtain a view control network.

[0022] In some embodiments, the teacher model represents a model capable of processing three-dimensional image information;

[0023] The teacher model is used to generate sample image features corresponding to the sample description information, including:

[0024] Using the teacher model, a three-dimensional image feature corresponding to the sample description information and a sample perspective control vector corresponding to the sample perspective information are generated; the sample perspective control vector represents a three-dimensional perspective control vector corresponding to the sample perspective information;

[0025] Based on the 3D image features, the teacher model is used to determine the sample image features corresponding to the sample view control vector.

[0026] In some embodiments, generating predicted image features corresponding to sample description information using a student model includes:

[0027] Adding random noise to the sample image feature corresponding to the sample description information to obtain a second image feature;

[0028] Based on the sample perspective information, a predicted perspective control vector is generated using a first neural network;

[0029] Based on the sample description information and the predicted view control vector, a denoising network is used to perform denoising processing on the second image feature to obtain a predicted image feature.

[0030] In some embodiments, determining loss information between the predicted image feature and the sample image feature corresponding to each sample description information includes:

[0031] For each sample description information, loss information is determined based on the predicted image features, sample image features and sample viewing angle control vector corresponding to the sample description information.

[0032] In some embodiments, determining loss information based on predicted image features, sample image features, and sample view control vectors corresponding to the sample description information includes:

[0033] Determine first noise distribution information of the teacher model for sample image features;

[0034] Determining second noise distribution information based on the predicted image features and the sample viewing angle control vector;

[0035] Based on the first noise distribution information and the second noise distribution information, loss information is determined.

[0036] In some embodiments, the image processing method further includes:

[0037] Based on the loss information, the denoising network in the student model is updated to obtain an updated student model as the text graph model.

[0038] On the other hand, the present application provides an image processing device, the device comprising:

[0039] An acquisition module, which acquires description information of the image to be generated and a first image feature; the description information includes viewing angle information corresponding to the image to be generated; and the first image feature represents an image feature containing noise information;

[0040] A perspective determination module determines a target perspective control vector based on the perspective information;

[0041] A denoising module performs denoising processing on the first image feature based on the description information and the target view control vector to obtain a target image feature;

[0042] The generation module generates a target image based on the target image features. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0044] Figure 1 A schematic diagram of an implementation flow of an image processing method provided in this application;

[0045] Figure 2 A schematic diagram of the implementation flow of another image processing method provided by the present application;

[0046] Figure 3 A schematic diagram of the implementation flow of another image processing method provided by the present application;

[0047] Figure 4 A schematic diagram of using a teacher model to guide a student model to perform model training according to the image processing method provided by the present application;

[0048] Figure 5 A schematic diagram of performing Wensheng graph reasoning using a Wensheng graph model determined by the image processing method provided by the present application;

[0049] Figure 6 A schematic diagram of the structure of an image processing device provided by the present application;

[0050] Figure 7 A hardware entity diagram of a computer device provided for this application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0052] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0053] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0055] In order to solve the problem of generating images from different perspectives based on text, related technologies simulate image transformations under different perspectives by pre-building a three-dimensional (3D) model or rendering a set of multi-perspective images. However, these methods are often limited by the quantity, quality and computational complexity of data, and it is difficult to accurately capture and reproduce the details and intentions in the text description.

[0056] Accordingly, the present application provides an image processing method, which can be executed by a processor of a computer device. The computer device may refer to a server, a laptop, a tablet computer, a desktop computer, a smart TV, a set-top box, a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, a portable gaming device), or other device with image processing capabilities.

[0057] Figure 1 A schematic diagram of the implementation process of an image processing method provided in this application is shown in FIG. Figure 1 As shown, the method includes the following steps S101 to S103:

[0058] Step S101, obtaining description information of an image to be generated and a first image feature; the description information includes viewing angle information corresponding to the image to be generated; the first image feature represents an image feature containing noise information.

[0059] Here, the description information refers to the text description information of the image to be generated when the model is used to generate the image.

[0060] In some embodiments, the description information may be determined based on the text information, voice information or video information input by the user. For example, the text information input by the user is used as the description information; for another example, the voice information input by the user is recognized to obtain the corresponding text information, and the text information is used as the description information; for another example, the video information input by the user is recognized to obtain the text information corresponding to the voice information therein and the text information corresponding to the action information therein, and the two parts of text information are used together as the description information, and so on.

[0061] In some embodiments, the description information may be text information obtained during the user's interaction with the interaction partner through a specified application, and the text image task is actively determined based on the text information. For example, during the user's social interaction through a social application, in response to the interaction information of the interaction partner about a specified scene, the description information of the image to be generated is determined.

[0062] The viewing angle information refers to the information about the image generation angle in the description information. For example, when the description information is "generate a side view of a cat", the viewing angle information is "side view".

[0063] In some embodiments, the viewing angle information may include observation point position information, viewing angle direction information, field of view angle size information, etc. corresponding to the object to be generated.

[0064] The first image feature refers to an initial image feature containing random noise.

[0065] In some embodiments, the random noise may include Gaussian Noise, Salt and Pepper Noise, and / or Garcia Noise, among others.

[0066] Step S102: determining a target viewing angle control vector based on the viewing angle information.

[0067] Here, the description information is encoded to obtain corresponding encoded data; and with respect to the encoded data corresponding to the viewing angle information, a target viewing angle control vector corresponding to the viewing angle information is determined.

[0068] In some embodiments, the target viewing angle control vector is a 3D viewing angle control vector corresponding to the image to be generated. Here, the 3D viewing angle control vector may describe the viewing angle information corresponding to the 2D image to be generated in the 3D space. In some embodiments, the 3D viewing angle control vector may be projected into a 2D viewing angle control vector to guide the generation effect of the 2D image to be generated.

[0069] In some embodiments, the target viewing angle control vector is a 2D viewing angle control vector corresponding to the image to be generated. Here, feature extraction is performed on the coded data corresponding to the viewing angle information to determine the corresponding 2D viewing angle control vector.

[0070] Step S103: Based on the description information and the target viewing angle control vector, denoising is performed on the first image feature to obtain a target image feature.

[0071] Here, based on the description information and the target viewing angle control vector, the noise signal included in the first image feature is predicted, and based on the predicted noise signal, denoising processing is performed on the first image feature to obtain the target image feature.

[0072] In some embodiments, based on the description information and the target view control vector, an attention mechanism is used to predict the noise signal from the first image feature.

[0073] Step S104: generating a target image based on the target image features.

[0074] Here, a decoding process is performed on the generated target image features to obtain the target image.

[0075] In the image processing method provided by the present application, by identifying the perspective information in the description information and determining the corresponding target perspective control vector, the target perspective control vector is used to influence the denoising process of the first image feature containing noise information, thereby obtaining a target image that can reflect the perspective information. In this way, in the process of generating an image using text, an image corresponding to the perspective in the description information can be generated based on the perspective information, thereby improving the accuracy of image generation and maintaining the consistency of perspective in the process of generating images with different perspectives, thereby improving the user experience.

[0076] In some embodiments, the determining of the target viewing angle control vector based on the viewing angle information, that is, the above step S102, can be implemented as the following steps S1021 to S1022:

[0077] Step S1021, determining a first viewing angle control vector corresponding to the viewing angle information; the first viewing angle control vector represents a three-dimensional vector used to control an image generation angle;

[0078] Step S1022: Project the first viewing angle control vector into a two-dimensional second viewing angle control vector, and use the second viewing angle control vector as the target viewing angle control vector.

[0079] Here, first, a neural network capable of processing 3D spatial information is used to extract features of the perspective information to obtain a 3D space vector for controlling the perspective of graphics generation, that is, a first perspective control vector; then, the first perspective control vector is converted to 2D space to obtain a 2D second perspective control vector; in this way, the second perspective control vector is used as a target perspective control vector to affect the generation process of a 2D target image.

[0080] In some embodiments, Figure 2 As shown, an image processing method provided by the present application includes the following steps S201 to S204:

[0081] Step S201, obtaining description information of an image to be generated and a first image feature; the description information includes viewing angle information corresponding to the image to be generated; the first image feature represents an image feature containing noise information.

[0082] Here, step S201 corresponds to the above step S101, so the specific implementation of step S201 can refer to the detailed description of the above step S101.

[0083] Step S202: Based on the viewing angle information, the target viewing angle control vector is determined by using the viewing angle control network in the Wensheng graph model.

[0084] Here, the text-to-graph model refers to a model used to generate 2D images from text.

[0085] In some embodiments, the Wensheng graph model is an improved diffusion model obtained after model training.

[0086] The perspective control network is a neural network used in the Wensheng graph model to generate a target perspective control vector based on perspective information.

[0087] In some embodiments, the process of generating a target perspective control vector by a perspective control network may include: first, generating a 3D first perspective control vector based on perspective information; then, converting the first perspective control vector into a 2D second perspective control vector, and outputting the second perspective control vector as a target perspective control vector.

[0088] Step S203: Based on the description information and the target viewing angle control vector, a denoising network in the text graph model is used to perform denoising processing on the first image feature to obtain the target image feature.

[0089] In some embodiments, when the Wensheng graph model is an improved diffusion model, the denoising network is a U-Net network in the improved diffusion model, wherein the U-Net network is used to predict the noise signal contained in the first image feature based on the description information and the target perspective control vector, and remove the predicted noise signal from the first image feature, thereby obtaining the denoised target image feature.

[0090] The classic U-Net network structure includes multiple encoders, intermediate connection parts and multiple decoders, each encoder and decoder includes a residual network and an attention network.

[0091] In some embodiments, a perspective control network is inserted into the attention network in at least one encoder and decoder to use the target perspective control vector output by the perspective control network to influence the data processing result of the attention network of the corresponding encoder or decoder, that is, to enable the U-Net network to pay attention to the target perspective control vector when performing noise prediction.

[0092] Step S204: Based on the target image features, the target image is generated using a decoder network in the text graph model.

[0093] Here, the decoder network is used to understand the target image features and generate the corresponding target image.

[0094] In some embodiments, when the Wensheng graph model is an improved diffusion model, in order to reduce the dimension of the image feature data processed by the diffusion model, thereby reducing the amount of data calculation of the diffusion model, the decoder part in the Variational Autoencoder (VAE) can be used as the decoder part of the improved diffusion model.

[0095] In the above embodiment, by inserting the perspective control network into the classic diffusion model, the improved diffusion model can identify the perspective information in the description information and generate the target perspective control vector based on the perspective information, so that the denoising network of the improved diffusion model can generate the target image features based on the description information and the target perspective control vector. In this way, the improved diffusion module can generate a consistent perspective image based on the perspective information in the description information, thereby improving the user experience; at the same time, compared with the classic diffusion model, the above-mentioned Wensheng graph model only adds the calculation process of the target perspective control vector through the perspective control network, so the overall calculation complexity is basically the same as the complexity of the classic diffusion model, that is, while realizing the generation of images with different perspectives, the calculation complexity of the model is not increased.

[0096] In some embodiments, Figure 3 As shown, the image processing method provided by the present application also includes the following steps S301 to S304:

[0097] Step S301, for each sample description information in the sample set, use the teacher model to generate the sample image features corresponding to the sample description information; each sample description information includes corresponding sample viewing angle information; the sample image features represent the two-dimensional image features corresponding to the sample description information at the sample viewing angle.

[0098] Here, the sample set represents a collection of multiple sample description information used to train the student model.

[0099] In some embodiments, each sample description information is text description information of the image to be generated.

[0100] Each sample description information includes sample viewing angle information for indicating the generation angle of the image to be generated. In some embodiments, the sample viewing angle information may include looking down, looking sideways or looking up and the corresponding specific angles, or the size of the field of view, etc.

[0101] The teacher model refers to a pre-trained model that can recognize sample perspective information in sample description information and generate a 2D image corresponding to the sample perspective information.

[0102] In some embodiments, the teacher model is a model that can process 3D spatial information. In this way, the teacher model can generate a 3D viewing angle control vector based on the sample viewing angle information, and determine the 2D image features corresponding to the sample description information based on the 3D viewing angle control vector.

[0103] In some embodiments, the teacher model can be a 3D Neural Radiance Fields (NeRF) model, a FastScene model, or other common models in the art that can realize text generation of 3D images. In this way, the teacher model can generate 3D image features based on the description information, and determine the 2D image features corresponding to the sample perspective information, i.e., the sample image features, from the generated 3D image features based on the sample perspective information and the camera projection principle.

[0104] Step S302: for each sample description information, using a student model to generate predicted image features corresponding to the sample description information; the student model includes a first neural network and the denoising network; the first neural network is used to generate a predicted perspective control vector based on the sample perspective information.

[0105] Here, the student model refers to a model trained under the guidance of the teacher model, and the student model after model training is the Wensheng graph model mentioned above.

[0106] In some embodiments, the student model is an improved diffusion model, that is, the student model is obtained by inserting the first neural network into the classical diffusion model.

[0107] Here, the first neural network is a perspective control network before model training, that is, an initial perspective control network.

[0108] In some embodiments, the first neural network is a neural network inserted into the attention network of the denoising network, which is used to identify sample perspective information in the sample description information and predict the corresponding target perspective control vector based on the sample perspective information.

[0109] In this way, after each sample description information is input into the student model, the denoising network in the student model can perform denoising processing on the image features containing random noise based on the sample description information and the target perspective control vector predicted by the first neural network to obtain the corresponding predicted image features.

[0110] Step S303: determining loss information between the predicted image feature and the sample image feature corresponding to each sample description information.

[0111] Here, the loss information is used to measure the error between the predicted image feature corresponding to each sample description information and the image feature of the sample image, or the error between the noise information contained in the predicted image feature and the noise information added to the corresponding sample image feature.

[0112] In some embodiments, a preset loss function is used to calculate the loss information between the predicted image feature corresponding to each sample description information and the sample image.

[0113] In some embodiments, the loss function may be established based on square loss, mean square error loss, or the like.

[0114] Step S304: based on the loss information, update the first neural network to obtain the viewing angle control network.

[0115] Here, the network parameters of the neural networks in the student model except the first neural network are frozen. Therefore, based on the loss information corresponding to each sample description information, the first neural network in the student model is updated so that the error between the predicted image features generated based on the predicted target view control information generated by the updated first neural network and the corresponding sample image features is smaller, thereby more accurately expressing the corresponding sample description information.

[0116] In some embodiments, the first neural network is updated multiple times by calculating the loss information between the predicted image features corresponding to the sample description information and the sample image until the calculated loss information is reduced to a specified loss threshold, thereby determining the first neural network after the last update as the corresponding perspective control network.

[0117] In some embodiments, the image processing method provided by the present application further includes: based on the loss information, updating the denoising network in the student model to obtain an updated student model as the Wensheng graph model.

[0118] Here, based on the calculated loss information, the first neural network and the denoising network of the student model are updated simultaneously. At the same time, when the first neural network is a part of the denoising network, the network parameters of the first neural network in the denoising network and the network outside the first neural network are updated based on the loss information, so that the overall update of the denoising network can be achieved.

[0119] In the above embodiment, first, the teacher model is used to generate a sample image corresponding to each sample description information; then, the student model is used to generate a predicted image feature corresponding to each sample description information; then, the loss information between the predicted image feature corresponding to each sample description information and the sample image feature is calculated; finally, the first neural network used to generate the predicted perspective control vector in the student model is updated based on the loss information. In this way, since the teacher model can accurately identify the sample perspective information in the sample description information and generate the sample image feature corresponding to the sample perspective, the sample description information and the sample image generated by the teacher model are used to guide the model training of the student model, so that the student model can correctly identify the sample perspective information in the sample description information and generate the predicted perspective control vector, thereby affecting the generation process of the predicted image feature, that is, in the training process of the student model, it is not necessary to rely on a large number of images with different perspectives, thereby reducing the requirements for the amount of data for model training; at the same time, the teacher model is a pre-trained model that can generate images with different perspectives, therefore, the sample image generated by the teacher model has a high degree of matching with the corresponding sample description information, and accordingly, the sample set used to train the student model has a high data quality, thereby achieving a better model training effect.

[0120] In some embodiments, the teacher model represents a model capable of processing three-dimensional image information.

[0121] Here, the teacher model can process three-dimensional image information, which means that the teacher model can generate 3D images based on the description information. For example, when the teacher model is a 3D NeRF model, the 3D NeRF model can generate image features in a 3D space based on the description information.

[0122] Thus, the generation of the sample image features corresponding to the sample description information by using the teacher model in the above step S301 can be implemented as the following steps S3011 to S3012:

[0123] Step S3011, using the teacher model, generate a three-dimensional image feature corresponding to the sample description information and a sample perspective control vector corresponding to the sample perspective information; the sample perspective control vector represents a three-dimensional perspective control vector corresponding to the sample perspective information.

[0124] Here, by utilizing the three-dimensional image information processing capability, the teacher model can generate corresponding 3D image features based on the sample description information; at the same time, the teacher model can identify the sample perspective information in the sample description information and generate the corresponding sample perspective control vector.

[0125] In some embodiments, the input network in the teacher model receives sample description information and performs encoding processing on the sample description information to obtain encoding information corresponding to the sample description information.

[0126] In some embodiments, the semantic analysis network of the teacher model is used to perform semantic analysis and understanding on the sample description information to extract key information such as the object, color, state, and perspective in the description information.

[0127] Here, the sample perspective control vector generated by the teacher model is a 3D vector, which is used to indicate the sample perspective information in the sample description information.

[0128] Step S3012: Based on the three-dimensional image features, use the teacher model to determine the sample image features corresponding to the sample viewing angle control vector.

[0129] Here, the teacher model projects the 3D image features according to the sample perspective control vector corresponding to the sample perspective information based on the camera projection principle, thereby obtaining the 2D image features corresponding to the sample perspective control vector.

[0130] In some embodiments, the 3D image feature is projected to a 2D image feature corresponding to the sample viewing angle control vector based on a perspective projection transformation or a parallel projection transformation.

[0131] In the above embodiment, the teacher model is used to generate 3D image features corresponding to the sample description information, and based on the camera projection principle, the 3D image features are projected into 2D image features corresponding to the sample view control vector as sample image features. In this way, in the absence of sufficient sample description information and corresponding 2D image pairs, the sample image features generated by the teacher model can be used as the true labels of the corresponding sample description information to be used as training samples for the student model.

[0132] In some embodiments, the step S302 of using the student model to generate the predicted image features corresponding to the sample description information may be implemented as the following steps S3021 to S3023:

[0133] Step S3021: adding random noise to the sample image feature corresponding to the sample description information to obtain a second image feature.

[0134] Here, for the sample image features generated by the teacher model, noise information is added according to a preset time step to obtain the second image features.

[0135] In some embodiments, the noise information added to the sample image features may be Gaussian noise, salt and pepper noise, Garcia noise and / or other random noise.

[0136] Step S3022: Based on the sample perspective information, use the first neural network to generate a predicted perspective control vector.

[0137] Here, the sample description information is input into the student model; the sample description information is encoded using the encoder of the student model to obtain corresponding encoded information; the sample perspective information in the sample description information is feature extracted using the first neural network of the student model to obtain a predicted perspective control vector corresponding to the sample perspective information.

[0138] In some embodiments, the first neural network generates a corresponding 3D predicted perspective control vector based on the sample perspective information, and transforms the 3D predicted perspective control vector into a 2D predicted perspective control vector to utilize the 2D predicted perspective control vector to influence the image generation process of the student model.

[0139] Step S3023: Based on the sample description information and the predicted viewing angle control vector, the denoising network is used to perform denoising processing on the second image feature to obtain the predicted image feature.

[0140] Here, based on the sample description information and the predicted view control vector, the denoising network of the student model is used to predict the noise information in the second image feature, and based on the second image feature and the predicted noise information, the predicted image feature is obtained.

[0141] In some embodiments, based on the sample description information and the predicted view control vector, the attention network in the denoising network is used to adjust the feature weights in the second image features, thereby predicting the corresponding noise information.

[0142] In some embodiments, the step S303 of determining the loss information between the predicted image feature and the sample image feature corresponding to each sample description information may be implemented as the following step S3031:

[0143] Step S3031: for each sample description information, determine the loss information based on the predicted image features, sample image features and sample viewing angle control vector corresponding to the sample description information.

[0144] Here, for each sample description information, the error between the predicted image features generated by the student model and the sample image features generated by the teacher model is calculated with the sample view control vector generated by the teacher model as the constraint condition.

[0145] In some embodiments, calculating the error between the predicted image feature and the sample image feature may refer to calculating the error between the predicted noise signal corresponding to the predicted image feature and the noise signal added to the sample image feature.

[0146] In this way, the predicted image features, sample image features and sample viewing angle control vectors corresponding to the sample description information in the above step S3031 can also be implemented as the following steps S3032 to S3034:

[0147] Step S3032, determining the first noise distribution information of the teacher model for the sample image features.

[0148] Here, the first noise distribution information is the noise information added to the sample image features by the teacher model. Therefore, the first noise distribution information is the real noise distribution information.

[0149] According to the preset time step, the teacher model adds noise information to the sample image feature multiple times to obtain multiple second image features containing different levels of noise. In this way, the sample image feature corresponds to multiple second image features, each of which has corresponding time step information and noise distribution information.

[0150] Step S3033: Determine second noise distribution information based on the predicted image features and the sample viewing angle control vector.

[0151] Here, based on the predicted image features generated by the student model and the sample perspective control vector generated by the teacher model, the second noise distribution information in the predicted image features is calculated, that is, the noise distribution information corresponding to the predicted image features is calculated under the restriction of the sample perspective control vector.

[0152] Step S3034: determine the loss information based on the first noise distribution information and the second noise distribution information.

[0153] Here, the error between the first noise distribution information (ie, the actual noise distribution information) and the predicted second noise distribution information is calculated to obtain the loss information.

[0154] In some embodiments, a specified loss function is used to calculate the loss information between the predicted image features and the sample image features. Here, the loss function is shown in the following formula (1):

[0155]

[0156] Among them, θ L V Indicates the calculated loss information;

[0157] t represents the time step information;

[0158] y represents the sample description information of the output teacher model and student model;

[0159] c represents the sample perspective control vector generated by the teacher model;

[0160] x t represents the image features obtained after the teacher model adds noise to the sample image features at time step t; x′ trepresents the predicted image features generated by the student model at time step t;

[0161] w(t) represents the preset weight function;

[0162] ε ψ (x t ,t,y) represents the noise distribution information added by the teacher model to the sample image at time step t;

[0163] ε′ ψ (x′ t ,t,y,c) represents the noise distribution information predicted by the student model at time step t;

[0164] θ represents the preset three-dimensional parameter used in the formula Calculate the partial derivatives in .

[0165] In this way, using the above loss function, the loss between the predicted image features and the sample image features is determined based on the noise distribution information added to the sample image features and the noise distribution information in the predicted image determined under the influence of the sample view control vector.

[0166] When calculating the loss information, the noise distribution information corresponding to the predicted image features is determined under the influence of the sample perspective control vector. Therefore, after updating the first neural network in the student model based on the loss information, the first neural network can more accurately establish the correspondence between the sample perspective information and the sample control vector, that is, the first neural network can predict a 3D predicted perspective control vector that is closer to the sample perspective control vector based on the sample perspective information, and then determine the corresponding 2D predicted perspective control vector based on the 3D predicted perspective control vector.

[0167] Next, combine Figure 4 , an example of using a teacher model to guide a student model to perform model training is described.

[0168] like Figure 4 As shown, first, the sample description information 401 and the third image feature 402 are input into the teacher model 410.

[0169] Here, the third image feature 402 is an image feature including random noise.

[0170] Sample description information 401 is description information of the image to be generated, that is, "a top view of a man and a woman sitting on a sofa"; wherein sample description information 401 includes sample perspective information "top view". Thus, sample description information 401 can be expressed as "generating an image of a man and a woman sitting on a sofa from a top view".

[0171] Here, the teacher model 410 is a 3D NeRF model. In this way, the teacher model 410 can generate a 3D image feature containing the information "a man and a woman sitting on a sofa"; at the same time, the teacher model 410 recognizes the sample perspective information "top view" and generates a corresponding sample perspective control vector 404; the teacher model 410 performs perspective processing on the generated 3D image feature based on the camera projection principle and the sample perspective control vector 404, and generates a 2D sample image feature 405 corresponding to the sample perspective control vector 404.

[0172] Then, the teacher model adds random noise to the 2D image feature 405 according to a preset time step to obtain a fourth image feature 406 containing noise information.

[0173] Afterwards, the fourth image feature 406 , the sample description information 401 , and the sample viewing angle control vector 404 are input into the student model 420 to perform model training on the student model 420 .

[0174] After the student model 420 receives the sample description information 401, it first encodes the sample description information 401; then, the first neural network 421 in the student model 420 generates a 3D predicted perspective control vector based on the sample perspective information in the sample description information 401, and converts the 3D predicted perspective control vector into a 2D predicted perspective control vector; then, the denoising network in the student performs denoising on the fourth image feature 406 based on the sample description information and the predicted perspective control vector to obtain the denoised predicted image feature 407.

[0175] Afterwards, based on the sample view control vector 404, the sample image feature 405 and the predicted image feature 407, the loss information is calculated using the loss function shown in the above formula (1), and the loss information is used to update the first neural network 421.

[0176] In this way, based on multiple sample description information containing sample perspective information, the teacher model is used to generate corresponding sample images and sample perspective control vectors, and then the first neural network in the student model is updated multiple times based on the multiple sample description information and the corresponding sample images and sample perspective control vectors to obtain the trained first neural network, that is, the perspective control network in the student model, and the trained student model is used as the Wensheng graph model.

[0177] Next, combine Figure 5 , an embodiment of performing a Vincent graph task using a Vincent graph model trained by the image processing method provided by the present application is described. The denoising network of the Vincent graph model is a U-Net network; and the viewing angle control network is a neural network inserted into the attention network of the U-Net network.

[0178] like Figure 5 As shown, first, the description information 501 is input into the text encoder 510 of the text graph model.

[0179] Here, the content of description information 501 is "a top view of a man and a woman sitting on a sofa", wherein the viewing angle information is "a top view".

[0180] The text encoder 510 encodes the description information 501 to obtain encoding information corresponding to the description information 501. The text encoder 510 sends the encoded description information 501 to the U-Net network 520 of the text graph model, and at the same time, generates a fifth image feature (not shown in the figure) containing random noise in the latent space of the text graph model, and inputs the fifth image feature into the U-Net network 520.

[0181] The perspective control network 521 in the U-Net network 520 generates a target perspective control vector based on the perspective information in the description information 501, so that the U-Net network 520 performs denoising on the fifth image feature based on the description information 501 of the target perspective control vector to obtain the denoised target image feature.

[0182] The U-Net network 520 outputs the target image features to the VAE decoder 530.

[0183] The VAE decoder 530 performs decoding processing on the target image features to obtain a target image 502 corresponding to the target image features.

[0184] In the above embodiment, the perspective control layer 521 in the U-Net network 520 of the Wensheng graph model is used to identify the perspective information in the description information 501 and generate a corresponding target perspective control vector, so that the target perspective control vector is used to influence the denoising process of the U-Net network 520 on the fifth feature image containing random noise, thereby generating a target image feature corresponding to the perspective information in the description information 501.

[0185] Based on the foregoing embodiments, the present application provides an image processing device, which includes the units included and the modules included in the units, which can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0186] Figure 6 A schematic diagram of the structure of an image processing device provided in this application is shown in FIG. Figure 6 As shown, the image processing device 600 includes: an acquisition module 610, a viewing angle determination module 620, a denoising module 630 and a generation module 640, wherein:

[0187] An acquisition module 610 acquires description information of the image to be generated and a first image feature; the description information includes viewing angle information corresponding to the image to be generated; the first image feature represents an image feature containing noise information;

[0188] A viewing angle determination module 620 determines a target viewing angle control vector based on the viewing angle information;

[0189] A denoising module 630 performs denoising processing on the first image feature based on the description information and the target viewing angle control vector to obtain a target image feature;

[0190] The generating module 640 generates a target image based on the target image features.

[0191] In some embodiments, the viewing angle determination module 620 is used to:

[0192] Determine a first viewing angle control vector corresponding to the viewing angle information; the first viewing angle control vector represents a three-dimensional vector used to control an image generation angle;

[0193] The first viewing angle control vector is projected into a two-dimensional second viewing angle control vector, and the second viewing angle control vector is used as the target viewing angle control vector.

[0194] In some embodiments, the perspective determination module 620 is used to determine the target perspective control vector based on the perspective information using a perspective control network in a Wensheng graph model;

[0195] The denoising module 630 is used to perform denoising processing on the first image feature based on the description information and the target viewing angle control vector using a denoising network in the Wensheng graph model to obtain the target image feature;

[0196] The generating module 640 is used to generate the target image based on the target image features and using the decoder network in the text graph model.

[0197] In some embodiments, the image processing apparatus 600 further includes:

[0198] The sample image generation module 650 is used to generate a sample image feature corresponding to each sample description information in the sample set by using a teacher model; each sample description information includes corresponding sample viewing angle information; the sample image feature represents a two-dimensional image feature corresponding to the sample description information at the sample viewing angle;

[0199] A predicted image generation module 660 is used to generate predicted image features corresponding to each sample description information using a student model; the student model includes a first neural network and the denoising network; the first neural network is used to generate a predicted perspective control vector based on the sample perspective information;

[0200] A loss calculation module 670, used to determine loss information between the predicted image feature corresponding to each of the sample description information and the sample image feature;

[0201] The updating module 680 is used to update the first neural network based on the loss information to obtain the viewing angle control network.

[0202] In some embodiments, the teacher model represents a model capable of processing three-dimensional image information;

[0203] The sample image generation module 650 is used to:

[0204] Using the teacher model, generating a three-dimensional image feature corresponding to the sample description information and a sample viewing angle control vector corresponding to the sample viewing angle information; the sample viewing angle control vector represents a three-dimensional viewing angle control vector corresponding to the sample viewing angle information;

[0205] Based on the three-dimensional image features, the teacher model is used to determine the sample image features corresponding to the sample viewing angle control vector.

[0206] In some embodiments, the predicted image generation module 660 is used to:

[0207] Adding random noise to the sample image feature corresponding to the sample description information to obtain a second image feature;

[0208] Based on the sample perspective information, using the first neural network to generate a predicted perspective control vector;

[0209] Based on the sample description information and the predicted viewing angle control vector, the denoising network is used to perform denoising processing on the second image feature to obtain the predicted image feature.

[0210] In some embodiments, the loss calculation module 670 is used to determine the loss information for each sample description information based on the predicted image features, sample image features and sample viewing angle control vector corresponding to the sample description information.

[0211] In some embodiments, the loss calculation module 670 is used to:

[0212] Determine first noise distribution information of the teacher model for the sample image feature;

[0213] Determining second noise distribution information based on the predicted image feature and the sample viewing angle control vector;

[0214] The loss information is determined based on the first noise distribution information and the second noise distribution information.

[0215] In some embodiments, the loss calculation module 670 is further used to update the denoising network in the student model based on the loss information to obtain an updated student model as the text graph model.

[0216] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0217] It should be noted that in the embodiment of the present application, if the above-mentioned image processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0218] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0219] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0220] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0221] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0222] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0223] It should be noted that Figure 7 A schematic diagram of a hardware entity of an electronic device in this application, such as Figure 7 As shown, the hardware entity of the electronic device 700 includes: a processor 701, a communication interface 702 and a memory 703, wherein:

[0224] The processor 701 generally controls the overall operation of the electronic device 700 .

[0225] The communication interface 702 enables the electronic device to communicate with other terminals or servers through a network.

[0226] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or processed by the processor 701 and each module in the electronic device 700 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be performed between the processor 701, the communication interface 702, and the memory 703 through the bus 704.

[0227] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0228] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0229] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0230] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0231] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0232] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0233] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0234] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire description information and a first image feature of an image to be generated; the description information includes viewing angle information corresponding to the image to be generated; the first image feature represents an image feature containing noise information; Based on the perspective information, determining a target perspective control vector; Based on the description information and the target viewing angle control vector, performing denoising processing on the first image feature to obtain a target image feature; Based on the target image features, a target image is generated.

2. The method according to claim 1, wherein: The determining a target viewing angle control vector based on the viewing angle information includes: Determine a first viewing angle control vector corresponding to the viewing angle information; the first viewing angle control vector represents a three-dimensional vector used to control an image generation angle; The first viewing angle control vector is projected into a two-dimensional second viewing angle control vector, and the second viewing angle control vector is used as the target viewing angle control vector.

3. The method according to claim 1 or 2, wherein: The determining the target perspective control vector based on the perspective information includes: determining the target perspective control vector based on the perspective information using a perspective control network in a Vincent graph model; The performing denoising processing on the first image feature based on the description information and the target viewing angle control vector to obtain the target image feature includes: performing denoising processing on the first image feature based on the description information and the target viewing angle control vector using a denoising network in the Vincent graph model to obtain the target image feature; The generating the target image based on the target image features includes: generating the target image based on the target image features by using a decoder network in the text graph model.

4. The method according to claim 3, further comprising: For each sample description information in the sample set, using the teacher model to generate a sample image feature corresponding to the sample description information; Each of the sample description information includes corresponding sample viewing angle information; the sample image feature represents a two-dimensional image feature corresponding to the sample description information at the sample viewing angle; For each sample description information, a student model is used to generate a predicted image feature corresponding to the sample description information; the student model includes a first neural network and the denoising network; the first neural network is used to generate a predicted viewing angle control vector based on the sample viewing angle information; Determine loss information between the predicted image feature and the sample image feature corresponding to each of the sample description information; Based on the loss information, the first neural network is updated to obtain the perspective control network.

5. The method according to claim 4, wherein: The teacher model represents a model capable of processing three-dimensional image information; The step of using the teacher model to generate sample image features corresponding to the sample description information includes: Using the teacher model, generating a three-dimensional image feature corresponding to the sample description information and a sample viewing angle control vector corresponding to the sample viewing angle information; the sample viewing angle control vector represents a three-dimensional viewing angle control vector corresponding to the sample viewing angle information; Based on the three-dimensional image features, the teacher model is used to determine the sample image features corresponding to the sample viewing angle control vector.

6. The method according to claim 5, wherein: The step of using the student model to generate predicted image features corresponding to the sample description information includes: Adding random noise to the sample image feature corresponding to the sample description information to obtain a second image feature; Based on the sample perspective information, using the first neural network to generate a predicted perspective control vector; Based on the sample description information and the predicted viewing angle control vector, the denoising network is used to perform denoising processing on the second image feature to obtain the predicted image feature.

7. The method according to claim 6, wherein: The determining of loss information between the predicted image feature and the sample image feature corresponding to each sample description information includes: For each of the sample description information, the loss information is determined based on the predicted image features, sample image features and sample viewing angle control vector corresponding to the sample description information.

8. The method according to claim 7, wherein: The determining the loss information based on the predicted image features, the sample image features and the sample viewing angle control vector corresponding to the sample description information includes: Determine first noise distribution information of the teacher model for the sample image feature; Determining second noise distribution information based on the predicted image feature and the sample viewing angle control vector; The loss information is determined based on the first noise distribution information and the second noise distribution information.

9. The method according to any one of claims 4, wherein: The method further comprises: Based on the loss information, the denoising network in the student model is updated to obtain an updated student model as the text graph model.

10. An image processing device, comprising: An acquisition module, which acquires description information of the image to be generated and a first image feature; the description information includes viewing angle information corresponding to the image to be generated; The first image feature represents an image feature containing noise information; A viewing angle determination module, which determines a target viewing angle control vector based on the viewing angle information; a denoising module, which performs denoising processing on the first image feature based on the description information and the target viewing angle control vector to obtain a target image feature; A generation module generates a target image based on the target image features.

Images