Image generation method and device, electronic equipment and storage medium

Abstract

The invention is suitable for the technical field of image generation, and provides an image generation method and device, electronic equipment and a storage medium, and the method comprises the steps: determining a target reference image according to a constraint condition, and the target reference image at least reflects a shooting angle requirement; performing encoding processing on the target reference image through a pre-trained image encoder to obtain an image code; and performing inverse diffusion processing on a noise image based on the constraint condition and the image code to obtain a target image. According to the invention, the image generation accuracy can be improved.

Description

Technical Field

[0001] The present application belongs to the field of image generation technology, and in particular, relates to an image generation method, device, electronic device and computer-readable storage medium. Background Art

[0002] AIGC (AI Generated Content) refers to the creation method of artificial intelligence-generated content, that is, through technologies such as machine learning and natural language processing, computers can automatically create multimedia content such as text, images, audio and even video. Among them, image generation is an important application field of AIGC technology.

[0003] Currently, in the process of using AIGC technology for image generation, when it is necessary to control the shooting angle of the generated image, text prompts are usually used to constrain the shooting angle of the generated image. However, the constraints of text prompts are usually weak, and it is difficult to accurately describe the shooting angle of the image in text descriptions. Therefore, it is difficult to accurately control the angle of the generated image during the image generation process. Summary of the invention

[0004] The embodiments of the present application provide an image generation method, an apparatus, an electronic device, and a storage medium, which can improve the accuracy of image generation.

[0005] In a first aspect, an embodiment of the present application provides an image generation method, comprising:

[0006] Determine a target reference image according to the constraint condition, wherein the target reference image at least reflects the shooting angle requirement;

[0007] Encoding the target reference image using a pre-trained image encoder to obtain an image code;

[0008] Based on the constraint conditions and the image coding, the noise image is subjected to inverse diffusion processing to obtain a target image.

[0009] In a possible implementation manner of the first aspect, before encoding the target reference image by using a pre-trained image encoder to obtain the image encoding, the method further includes:

[0010] Acquire a plurality of training samples with different shooting angles, wherein labels of the training samples at least include shooting parameters, and the shooting parameters at least include the shooting angle;

[0011] The training sample is encoded by the constructed image encoder to obtain a sample code;

[0012] Decoding the sample code to obtain a predicted value corresponding to the label;

[0013] The parameters of the image encoder are updated according to the difference between the predicted value and the true value corresponding to the label until the updated image encoder meets the training requirements, thereby obtaining the pre-trained image encoder.

[0014] In a possible implementation manner of the first aspect, the label of the training sample further includes a ground position and a parallel line position, and the updating of the parameters of the image encoder according to the difference between the predicted value and the true value corresponding to the label until the updated image encoder meets the training requirements to obtain the pre-trained image encoder includes:

[0015] For each of the labels of the training samples, a local loss value is calculated according to the true value corresponding to the label, the predicted value and the loss function corresponding to the label, and the restored value corresponding to the label is determined according to the restored image;

[0016] Determine a global loss value according to each of the local loss values;

[0017] The parameters of the image encoder are updated according to the global loss value until the updated image encoder converges to obtain the pre-trained image encoder.

[0018] In a possible implementation manner of the first aspect, determining the target reference image according to the constraint condition includes:

[0019] In a case where the constraint condition does not include the target reference image, determining a shooting parameter requirement according to the constraint condition, the shooting parameter requirement at least including the shooting angle requirement;

[0020] The target reference image is determined according to the shooting parameter requirement.

[0021] In a possible implementation manner of the first aspect, determining the target reference image according to the shooting parameter requirement includes:

[0022] Determining, according to the shooting parameter requirement, a candidate reference image that meets the shooting parameter requirement;

[0023] When the number of the candidate reference images is greater than 1, the target reference image is determined from each of the candidate reference images according to the degree of matching between the candidate reference images and the shooting parameter requirements, or the target reference image is determined according to a selection instruction, and the selection instruction is used to indicate the candidate reference image selected from each of the candidate reference images.

[0024] In a possible implementation manner of the first aspect, determining, according to the shooting parameter requirement, a candidate reference image that meets the shooting parameter requirement includes:

[0025] In a case where the accuracy required by the shooting parameter meets the precision requirement, generating the candidate reference image according to the shooting parameter requirement;

[0026] In the case that the accuracy required by the shooting parameters does not meet the precision requirement, the image that meets the shooting parameter requirement is screened out from various pre-stored images to obtain the candidate reference image.

[0027] In a second aspect, an embodiment of the present application provides an image generating device, including:

[0028] A target reference image acquisition module, used to determine a target reference image according to a constraint condition, wherein the target reference image at least reflects a shooting angle requirement;

[0029] An encoding module, used for encoding the target reference image through a pre-trained image encoder to obtain an image code;

[0030] A generating module is used to perform inverse diffusion processing on the noise image based on the constraint condition and the image coding to obtain a target image.

[0031] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the image generation method described in the first aspect when executing the computer program.

[0032] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image generation method described in the first aspect are implemented.

[0033] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the image generation method described in the first aspect above.

[0034] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0035] In the embodiment of the present application, after the constraint condition is obtained, the target reference image is first determined according to the constraint condition, and then the image encoding of the target reference image is obtained by encoding through an image encoder. Since the target reference image in the form of an image can reflect the shooting angle requirements more intuitively and accurately than a text description, that is, the image encoding in the form of a vector can more accurately reflect the shooting angle requirements for the target image to be generated, therefore, in the process of reverse diffusion of the noise image according to the image encoding and the constraint condition, the shooting angle of the image can be better constrained through the image encoding in the form of a vector, and the shooting angle of the generated image can be more accurately controlled, thereby improving the accuracy of the target image obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0037] Figure 1 It is a flowchart of an image generation method provided by an embodiment of the present application;

[0038] Figure 2 is a flowchart of another image generation method provided in an embodiment of the present application;

[0039] Figure 3 is a schematic diagram of the training process of the image encoder provided in an embodiment of the present application;

[0040] Figure 4 is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0041] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0043] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0044] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0045] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0046] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0047] Embodiment 1:

[0048] Figure 1 A schematic flow chart of an image generation method provided by an embodiment of the present invention is shown, and is described in detail as follows:

[0049] Step S101, determining a target reference image according to a constraint condition, wherein the target reference image at least reflects a shooting angle requirement.

[0050] The above-mentioned constraint conditions refer to information used to guide and limit the generation process to ensure that the generated image meets specific standards or requirements. In the embodiment of the present application, the acquired constraint conditions can be data in text form, image form and / or graphic form.

[0051] In some embodiments, the shooting angle may include a shooting direction and / or a shooting height, and correspondingly, the shooting angle requirement may include a shooting direction requirement and / or a shooting height requirement.

[0052] Since the shooting angle will have a great impact on the composition of the image and the expression of emotional atmosphere, it is usually difficult to accurately describe the shooting effect of the required shooting angle through words, and the constraints of words on image generation are often weak. Therefore, in order to ensure the accuracy of image generation, after obtaining the constraints of image generation, you can first determine the target reference image that can at least reflect the shooting angle requirements based on the constraints. Reflecting the shooting angle requirements through images can avoid ambiguity caused by unclear descriptions of the shooting angle requirements through words, compared to reflecting the shooting angle requirements through words. Thereby, the shooting angle of the generated image can be better constrained, and the controllability of the shooting angle during the image generation process can be improved.

[0053] Step S102: encoding the target reference image using a pre-trained image encoder to obtain an image code.

[0054] Optionally, the above-mentioned image encoder can be an encoder composed of a complex structure such as Variational Autoencoders (VAEs), Convolutional Autoencoders (Convolutional Autoencoders) or U-shaped Network (U-Net).

[0055] In some embodiments, in order to further improve the accuracy of image generation, the image encoder may encode the target reference image based on an attention mechanism such as self-attention or cross-attention, so as to better embed information such as the shooting angle in the target reference image into the obtained image coding, that is, to enable the image coding to more accurately reflect constraints such as its shooting angle requirements.

[0056] Step S103, performing inverse diffusion processing on the noise image based on the above constraint conditions and the above image coding to obtain a target image.

[0057] The noise image may be a fixed noise image or a random noise image generated based on noise such as Gaussian noise or random noise.

[0058] The above-mentioned inverse diffusion processing is the reverse diffusion processing. In the diffusion processing of an image, noise is usually gradually added to the image until the image is diffused into a pure noise, and the reverse diffusion processing is to gradually remove noise from the noise to restore the desired image, and the constraint condition is used to constrain the restored image in the process of removing noise from the noise to restore the image, so that the final image meets the constraint condition.

[0059] Among them, considering the influence of the constraints of text and the accuracy of description when describing the shooting angle requirements of the image through text, the target reference image that can reflect the shooting angle requirements is first determined according to the constraints. When performing the inverse diffusion process, the restored image will be constrained according to the constraints and the image code corresponding to the target reference image, so that the final image can better meet the shooting angle requirements. It should be noted that when the determined target reference image reflects all the requirements in the constraints, the noise image can be directly inversely diffused according to the image code to obtain the desired target image.

[0060] Optionally, the result of inverse diffusion processing of the noise image based on constraints and image coding can be an image, that is, the target image is directly obtained by inverse diffusion processing of the noise image based on constraints and image coding; or, the result of inverse diffusion processing of the noise image based on constraints and image coding can also be a latent space image coding. At this time, a pre-trained decoder can be used to decode the latent space image coding into an image to obtain the desired target image.

[0061] It should be noted that if the constraints include constrained text (i.e., constrained data in text form), the text data can be encoded using a pre-trained text encoder to obtain a text encoding in vector form, and then the noisy image can be inversely diffused based on the vector text encoding and image encoding.

[0062] In some embodiments, the text encoding and image encoding can be fused through an attention mechanism such as cross attention, so that the text information and shooting information such as the shooting angle in the image can interact effectively to obtain a fused encoding with higher accuracy, so that the subsequent image generation can be better guided according to the fused encoding, which is beneficial to improving the accuracy of the target image obtained.

[0063] In the embodiment of the present application, after the constraint conditions are obtained, the target reference image is first determined according to the constraint conditions, and then the image code of the target reference image is obtained by encoding through an image encoder. Since the target reference image in the form of an image can more intuitively and accurately reflect the shooting effect corresponding to the shooting angle than a text description, that is, the image code in the form of a vector can more accurately reflect the shooting angle requirements for the target image to be generated, therefore, the reliability of the shooting angle in the image generation process can be improved by using the image code in the form of a vector. When the noise image is inversely diffused according to the image code and the constraint conditions, the shooting angle of the generated image can be more accurately controlled, thereby improving the accuracy of the target image obtained.

[0064] In some embodiments, before the above step S102, the method further includes:

[0065] A1. Acquire a plurality of training samples with different shooting angles, wherein the labels of the training samples at least include shooting parameters, and the shooting parameters at least include the shooting angle.

[0066] Optionally, when obtaining training samples at different shooting angles, one or more scenes may be built or loaded in Unreal Engine (UE). After determining multiple sets of shooting parameters according to different shooting angles, shooting is performed in the scene according to each set of shooting parameters to obtain multiple training samples at different shooting angles.

[0067] Optionally, when determining multiple groups of shooting parameters according to different shooting angles, the multiple groups of shooting parameters may also be determined in combination with different camera intrinsic parameters and camera focal length and other parameters.

[0068] A2. Encode the above training samples through the constructed image encoder to obtain sample codes.

[0069] A3. Decode the sample code to obtain the predicted value corresponding to the label.

[0070] A4. Update the parameters of the image encoder according to the difference between the predicted value and the true value corresponding to the label, until the updated image encoder meets the training requirements, thereby obtaining the pre-trained image encoder.

[0071] After obtaining the training sample, the constructed image encoder is first used to encode the training sample to obtain the sample code, and then the sample code is decoded to predict the predicted value corresponding to the label of the sample image. Since the predicted value is predicted based on the image code of the sample image, the difference between the predicted value and the true value corresponding to the label of the sample image can reflect the accuracy of the encoded image code. Therefore, updating the parameters of the image encoder based on the difference between the true value of the label and its predicted value can better improve the accuracy of the image encoder, so that the image code obtained by encoding can better reflect the characteristics of the sample image, that is, the image encoder can better encode the shooting angle and other features of the sample image, thereby ensuring the accuracy of the pre-trained image encoder.

[0072] In the embodiment of the present application, the image encoder is trained using training samples with different shooting angles, so that the image encoder can fully learn the characteristics of the training samples with different shooting angles to improve the encoding accuracy of the pre-trained image encoder. At the same time, during the training process, the parameters of the image encoder are updated according to the difference between the predicted value obtained by decoding the sample encoding and the corresponding true value to improve the accuracy of the image encoding obtained by encoding, that is, to further improve the encoding accuracy of the obtained pre-trained image encoder.

[0073] In some embodiments, the labels of the training samples further include ground positions and parallel line positions.

[0074] To further improve the encoding accuracy of the image encoder, multiple features (target features) related to the shooting angle can be combined, labels corresponding to each target feature can be set for the training samples, and the true values ​​corresponding to each label of the training samples can be determined based on the training samples.

[0075] In the process of training the image encoder, the true value and predicted value corresponding to each label of the training sample can be combined to calculate the loss value corresponding to each target feature (i.e., each label), so as to update the parameters of the image encoder according to the loss value corresponding to each label, so that the final image encoder can better extract the various target features in the image.

[0076] In the embodiment of the present application, the labels of the training samples include shooting parameters, ground positions and parallel line positions. Correspondingly, the true values ​​corresponding to the respective labels include true shooting parameters, true ground positions and true parallel line positions.

[0077] Correspondingly, the above step A4 includes:

[0078] A41. For each of the labels of the training samples, a local loss value is calculated according to the true value corresponding to the label, the predicted value and the loss function corresponding to the label.

[0079] A42. Determine the global loss value based on the above local loss values.

[0080] A43. Update the parameters of the image encoder according to the global loss value until the updated image encoder converges to obtain the pre-trained image encoder.

[0081] In order to improve the accuracy of the image encoder, when training the image encoder, a multi-task learning method can be used to train the image encoder, so that the image encoder can effectively learn the intrinsic connections between these tasks and the common feature representations that are beneficial to each task in the process of multi-task joint learning, which is conducive to improving the encoding accuracy of the final pre-trained graphic encoder.

[0082] Among them, the embodiment of the present application adopts the three-task learning method of shooting parameter prediction, ground position prediction and parallel line position prediction to train the image encoder. The above-mentioned shooting parameter prediction is to predict the shooting parameters of the image, the above-mentioned ground position prediction refers to predicting the position of the ground plane in the image, and the above-mentioned parallel line position prediction refers to predicting the position of the parallel lines existing in the image. Since the shooting parameters affect the composition of the entire image, and the ground plane can better reflect the shooting height corresponding to the image, the vanishing point of the parallel lines can better reflect the shooting parameters such as the intrinsic and extrinsic parameters of the camera, the actual size and position of the objects in the scene, and the perspective type of the scene. Therefore, through the learning of the above-mentioned tasks, the pre-trained image encoder can better extract the features related to the shooting parameters, the ground and the parallel lines in the image, while effectively learning the internal connection between these features, further improving the accuracy of the encoded image encoding, which is conducive to improving the accuracy of image generation.

[0083] like Figure 2 As shown, after obtaining the predicted values ​​corresponding to each label according to the sample encoding and decoding, the predicted shooting parameters, predicted ground position and predicted parallel line position are determined according to the sample encoding, and then, for each label, the local loss value corresponding to the label is determined according to its corresponding true value and predicted value and the loss function. For example, for the ground position, the local loss value of the ground position is calculated according to the difference between the true ground position and the predicted ground position corresponding to the training sample and the loss function corresponding to the ground position.

[0084] Optionally, the loss functions corresponding to the various labels may be the same loss function or different loss functions, which may be specifically set according to the actual application scenario.

[0085] In some embodiments, a prediction network corresponding to each label may be set, and the sample code may be decoded by the prediction network corresponding to each label to obtain the prediction value corresponding to each label. Figure 3 As shown, a shooting parameter prediction network, a ground position prediction network and a parallel line position prediction network can be set, and each prediction network independently performs its corresponding decoding processing to obtain a prediction value corresponding to the prediction network.

[0086] In other embodiments, the labels of the training samples may also be shooting parameters, shooting heights, and vanishing point positions, and the true values ​​corresponding to the labels include true shooting parameters, true shooting heights, and true vanishing point positions. Correspondingly, when the predicted values ​​corresponding to the labels are determined according to the sample code, during the decoding process, for the shooting parameters, the predicted shooting parameters can be obtained according to the image code prediction; for the shooting height, the ground position in the image code can be detected first, and then the predicted shooting height can be predicted according to the ground position prediction; for the vanishing point position, the parallel lines in the image code can be detected first, and then the predicted vanishing point position can be predicted according to the position of the parallel lines.

[0087] In some embodiments, when determining the global loss value based on each local loss value, the local loss value can be weighted using the loss weight corresponding to each local loss value, and then the final global loss value is determined based on the weighted local loss values.

[0088] Optionally, the loss weight corresponding to each label can be determined in combination with the importance of different labels (i.e., different target features). For example, the more important the label, the greater the corresponding loss weight. Optionally, the importance of the label can be set by the user, or can be determined based on the degree of influence of each label on the composition of the image.

[0089] For example, since the shooting parameters directly affect the composition of the image, it can be considered that the shooting parameters have the greatest impact, that is, the shooting parameters are of the greatest importance, and their loss weight can be set to 0.5; the height of the camera above the ground will affect the field of view, detail expression and occlusion problems of the scene in the image, so it can be considered that the ground position label related to the height of the camera above the ground is second in importance, and its loss weight can be set to 0.35; the vanishing point will affect the perspective realism and aesthetics of the image, etc., and it can be considered that the parallel line position label related to the vanishing point is the least important, and its loss weight can be set to 0.15.

[0090] After obtaining the global loss value, the parameters of the image encoder can be updated by combining algorithms such as gradient descent until the updated image encoder converges to obtain the required pre-trained image encoder.

[0091] In an embodiment of the present application, the image encoder is trained by multi-task learning, so that the image encoder can effectively learn the intrinsic connections between different tasks and beneficial common feature representations during the learning process, which is beneficial to improving the encoding accuracy of the image encoder.

[0092] In some embodiments, the above step S101 includes:

[0093] B1. When the above constraint conditions do not include the above target reference image, determine shooting parameter requirements according to the above constraint conditions, and the above shooting parameter requirements at least include the above shooting angle requirements.

[0094] B2. Determine the target reference image according to the above shooting parameter requirements.

[0095] Since the acquired constraints may describe the shooting angle requirements in words, the required target reference image cannot be directly obtained from the constraints at this time. Therefore, in order to ensure that the shooting angle requirements can be better reflected through images in the subsequent image generation process, when the constraints do not include the target reference image, the target reference image that can at least reflect the shooting angle requirements can be determined based on the constraints, and then subsequent encoding and other processing can be performed.

[0096] Among them, since the constraints may also include requirements for other shooting-related parameters such as sensitivity and focal length, and these shooting-related parameters usually have a certain impact on the image, before determining the target reference image based on the constraints, the shooting parameter requirements can be first determined according to the requirements related to the shooting parameters in the constraints (such as shooting angle, camera position and rotation direction, etc.), and then the target reference image that meets the shooting parameter requirements can be determined based on the determined shooting parameter requirements, thereby fully ensuring the comprehensiveness and accuracy of the obtained target reference image.

[0097] In an embodiment of the present application, when the acquired constraint conditions describe the shooting angle requirements in a non-image manner such as text, the shooting parameter requirements including at least the shooting angle requirements are determined based on the constraint conditions, and then a target reference image that can better reflect the shooting angle requirements is obtained based on the shooting parameter requirements, so as to ensure that the shooting angle of the generated image can be better constrained through the target reference image with good performance in the subsequent process.

[0098] In some embodiments, the above step B2 includes:

[0099] B21. Determine a candidate reference image that meets the above shooting parameter requirements according to the above shooting parameter requirements.

[0100] B22. When the number of the candidate reference images is greater than 1, the target reference image is determined from each of the candidate reference images according to the degree of matching between the candidate reference images and the shooting parameter requirements, or the target reference image is determined according to a selection instruction, and the selection instruction is used to indicate the candidate reference image selected from each of the candidate reference images.

[0101] Optionally, the above-mentioned matching degree may be determined by manual analysis, or may be determined based on the similarity between the shooting parameter requirements and the candidate reference images (such as cosine similarity). In some embodiments, reference shooting parameters corresponding to each candidate reference image may be generated based on the candidate reference images, that is, the shooting parameters of each candidate reference image may be converted into text form, and then the shooting parameter requirements in text form may be compared with each reference shooting parameter to obtain the matching degree corresponding to each candidate reference image.

[0102] Optionally, the selection instruction may be an instruction manually input by the user, or an instruction generated according to a click operation of the user. For example, the user may manually input a selection instruction (e.g., "select the third candidate reference image") on a client device (e.g., a mobile phone), and after the server receives the selection instruction sent by the client device, it may directly determine the selected candidate reference image according to the selection instruction.

[0103] As an example, the above-mentioned image generation method is applied to an electronic device, and the selection instruction may be an instruction generated by a user pressing a preset button of the electronic device, and the generated selection instruction includes the identification information of the preset button (e.g., "button 3"). For example, assuming that the following five candidate reference images are obtained: image 1, image 2, image 3, image 4, and image 5; assuming that the user presses the preset button "button 4" of the electronic device, a selection instruction including the identification information "button 4" can be obtained. According to the selection instruction, it can be determined that the candidate reference image selected by the user is: image 4, that is, the target reference image is image 4.

[0104] As an example, the emotional atmosphere and other scene information of the target image to be generated can be analyzed according to the constraints and the actual application scenario to obtain the scene requirements, and then the target reference image can be screened out from the candidate reference images according to the degree of matching between the scene requirements and the candidate reference images, so that the determined target reference image is more in line with the actual application scenario, thereby improving the accuracy of the shooting parameter requirements reflected by the target reference image.

[0105] Since the shooting parameter requirements may not cover all shooting-related parameters, and are affected by various factors such as other parameters and light, multiple different candidate reference images may be determined according to the same shooting parameter requirements. Therefore, to ensure the accuracy of the acquired target reference image, if at least two candidate reference images that meet the shooting parameter requirements are determined, the best candidate reference image can be screened out from the multiple candidate reference images and used as the target reference image. The best candidate reference image can be determined according to the screening method adopted.

[0106] In an embodiment of the present application, each candidate reference image can be screened according to the degree of match between the candidate reference image and the shooting parameter requirements. In this case, the optimal candidate reference image (i.e., the target reference image) may be the candidate reference image with the highest degree of match; or, each candidate reference image can be screened according to a selection instruction. In this case, the optimal candidate reference image (i.e., the target reference image) may be the selected candidate reference image indicated by the selection instruction.

[0107] In an embodiment of the present application, when the number of candidate reference images that meet the shooting parameter requirements is greater than 1, the target reference image can be obtained by screening according to the degree of matching between the candidate reference images and the shooting parameter requirements, so that the determined target reference image is the image that best meets the shooting parameter requirements, or, the target reference image can be determined according to the selection instruction, so that the determined target reference image is an image that better meets the scene requirements or user requirements, thereby ensuring the accuracy of the target image finally obtained.

[0108] In some embodiments, the above step B21 includes:

[0109] In the case that the accuracy required by the above shooting parameters meets the precision requirement, the above candidate reference image is generated according to the above shooting parameter requirements.

[0110] In the case that the accuracy required by the shooting parameters does not meet the precision requirement, the images meeting the shooting parameter requirement are screened out from various pre-stored images to obtain the candidate reference images.

[0111] The above degree of accuracy is used to reflect the quality required by the shooting parameters.

[0112] Optionally, the above-mentioned precision can be determined according to the detail and / or accuracy of the shooting parameter requirements. The above-mentioned detail is usually determined by the number of parameters covered by the shooting parameter requirements. For example, the more parameters covered by the shooting parameter requirements, the more details covered, and the higher the detail. The above-mentioned accuracy is usually determined by the authenticity and reliability of each parameter in the shooting parameter requirements.

[0113] In some embodiments, the accuracy of the shooting parameter requirement may be determined according to the number of parameters covered by the shooting parameter requirement and the accuracy of each parameter.

[0114] For example, when the number of parameters covered by the shooting parameter requirement is less than a first number (such as 2), or the accuracy of each parameter is less than a first threshold (such as 0.7), the accuracy of the shooting parameter requirement can be considered to be low.

[0115] When the number of parameters covered by the shooting parameter requirements is greater than or equal to the first number and less than the second number (such as 5, which is greater than the first number), the accuracy of the shooting parameter requirements can be considered to be medium accuracy. Alternatively, when the accuracy of each parameter in the shooting parameter requirements is greater than or equal to the first threshold, the accuracy of the shooting parameter requirements can be considered to be medium accuracy.

[0116] When the number of parameters covered by the shooting parameter requirements is greater than or equal to the second number, or the accuracy of each parameter is greater than or equal to the second threshold (such as 0.9, greater than the first threshold), the accuracy of the shooting parameter requirements can be considered to be high accuracy.

[0117] When acquiring candidate reference images that meet the shooting parameter requirements, if the accuracy of the shooting parameter requirements meets the precision requirements (such as the accuracy is medium or high), it indicates that the shooting parameter requirements are of high quality and cover more details or the parameters have high accuracy. At this time, since the shooting parameter requirements can more accurately define the shooting parameters of the image, one or more candidate reference images that are more consistent with the shooting parameter requirements can be accurately generated through image generation processing, without generating a large number of candidate reference images with lower accuracy, thereby ensuring the accuracy of the obtained candidate reference images and helping to improve the efficiency of acquiring the target reference image.

[0118] If the accuracy required by the shooting parameters does not meet the precision requirements, the images that meet the shooting parameter requirements can be directly screened out from the pre-stored images according to the shooting parameter requirements to obtain the required candidate reference images without performing image generation processing, thereby avoiding the generation of a large number of candidate reference images with low accuracy. Optionally, the pre-stored images can be generated according to commonly used scenes, or the pre-stored images can also be images determined according to a training data set of an image encoder.

[0119] In an embodiment of the present application, based on whether the accuracy of the shooting parameter requirements meets the precision requirements, a suitable method is selected to obtain candidate reference images that meet the shooting parameter requirements, while ensuring the accuracy of the obtained candidate reference images and improving the efficiency of obtaining the candidate reference images as much as possible.

[0120] It should be noted that the image encoder corresponding to the encoding process involved in the above-mentioned image generation method and the encoder or network corresponding to the inverse diffusion process and the like can be integrated in the same model. When image generation is required, after the target reference image is determined according to the constraints, the constraints and the target reference image are used as the input of the trained integrated model. After the integrated model encodes the target reference image to obtain the image code, the noise image is inversely diffused according to the image code and the constraints to obtain the desired target image. In some embodiments, the image encoder in the integrated model is additionally trained using the method described in steps A1 to A4 above. After the integrated model is constructed according to the pre-trained image encoder, the parameters of the image encoder in the integrated model will not be modified when the integrated model is trained to obtain the trained integrated model.

[0121] For example, assuming that the above-mentioned image generation method is applied to the field of image reconstruction, after obtaining a pre-trained image encoder according to the above-mentioned steps A1 to A4, an image reconstruction model can be constructed based on the pre-trained image encoder, and the image reconstruction model may include an image encoder, a text encoder and an inverse diffusion network.

[0122] When training the constructed image reconstruction model, multiple training samples can be obtained, and the training samples can include a standard reference image (at least reflecting the shooting angle requirements), constraints and a standard reconstructed image. The standard reference image and the constraints are used as inputs of the constructed image reconstruction model to obtain a generated image output by the image reconstruction model. After obtaining the generated image, the parameters of the inverse diffusion network and the text encoder in the image reconstruction model can be adjusted according to the difference between the generated image and the corresponding standard reconstructed image until the image reconstruction model meets the requirements (such as the accuracy reaches a threshold of 90%), and a pre-trained image reconstruction model is obtained.

[0123] When generating an image, it is assumed that the target reference image and constraint text (i.e., the constraint condition includes the target reference image and the constraint text) input by the user through the target terminal are obtained, and the obtained constraint text and target reference image are used as inputs of the image reconstruction model. The image reconstruction model performs encoder processing on the target reference image through an image encoder to obtain image encoding, and performs encoding processing on the constraint text through a text encoder to obtain text encoding. Finally, the noise image is inversely diffused according to the image encoding and the text encoding to obtain the target image. After obtaining the target image, the target image can be sent to the target terminal for display.

[0124] It should be noted that the constraints obtained can be determined according to the actual application scenario. For example, assuming that the above-mentioned image generation method is applied to the super-resolution reconstruction task in image reconstruction, the super-resolution reconstruction task requires reconstructing a high-resolution image of a target object at a specific shooting angle according to a low-resolution image including the target object (i.e., a high-resolution image of the target object taken according to the specific shooting angle). In an embodiment of the present application, the user inputs the image to be reconstructed (including the target object) and the shooting angle requirement in text form (i.e., the constraints include the image to be reconstructed and the shooting angle requirement) through an input device (such as a mobile phone, etc.).

[0125] After obtaining the image to be reconstructed and the shooting angle requirement input by the user, a high-resolution image that meets the shooting angle requirement is first obtained according to the shooting angle requirement to obtain a target reference image, and then the target reference image, the image to be reconstructed and the text (i.e., the shooting angle requirement in text form) are input into the trained super-resolution reconstruction model. The super-resolution reconstruction model encodes the target reference image and the image to be reconstructed respectively through an image encoder to obtain a reference image code corresponding to the target reference image and a reconstructed image code corresponding to the image to be reconstructed, and the shooting angle requirement in text form is encoder processed through a text encoder to obtain a text code. Finally, the inverse diffusion network performs inverse diffusion processing on the noise image according to the reference image code, the reconstructed image code and the text code to obtain a high-resolution image containing the target object and meeting the shooting angle requirement.

[0126] For another example, suppose the above-mentioned image generation method is applied to a text image task. In this case, the constraint condition is a constraint text that at least includes shooting angle requirements. After determining the target reference image according to the acquired constraint conditions, the constraint text can be encoded by a pre-trained text encoder to obtain a text encoding, and the target reference image can be encoded by a pre-trained image encoder to obtain an image encoder. Finally, the noise image is inversely diffused according to the text encoding and the image encoding to obtain a target image that meets the constraint text. In addition, in the process of generating the target image, the generation of the image is constrained according to the shooting angle requirements in the text form and the image form. This can further improve the controllability of the image generation process, effectively solve the problem of poor text constraints, and improve the accuracy of the text image.

[0127] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0128] Embodiment 2:

[0129] Corresponding to the image generation method described in the above embodiment, Figure 4 A structural block diagram of an image generating device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0130] Reference Figure 4 The device comprises: a target reference image acquisition module 41, an encoding module 42 and a generation module 43.

[0131] A target reference image acquisition module 41 is used to determine a target reference image according to a constraint condition, wherein the target reference image at least reflects a shooting angle requirement;

[0132] The encoding module 42 is used to encode the target reference image through a pre-trained image encoder to obtain an image code;

[0133] The generating module 43 is used to perform inverse diffusion processing on the noise image based on the above constraint conditions and the above image coding to obtain a target image.

[0134] In the embodiment of the present application, after the constraint conditions are obtained, the target reference image is first determined according to the constraint conditions, and then the image code of the target reference image is obtained by encoding through an image encoder. Since the target reference image in the form of an image can more intuitively and accurately reflect the shooting effect corresponding to the shooting angle than a text description, that is, the image code in the form of a vector can more accurately reflect the shooting angle requirements for the target image to be generated, therefore, the reliability of the shooting angle in the image generation process can be improved by using the image code in the form of a vector. When the noise image is inversely diffused according to the image code and the constraint conditions, the shooting angle of the generated image can be more accurately controlled, thereby improving the accuracy of the target image obtained.

[0135] In some embodiments, the image generating device further includes:

[0136] A plurality of training samples with different shooting angles are obtained, wherein labels of the training samples at least include shooting parameters, and the shooting parameters at least include the shooting angle.

[0137] The sample encoding module is used to encode the above training samples through the constructed image encoder to obtain sample codes.

[0138] The decoding module is used to decode the sample code to obtain the predicted value corresponding to the label.

[0139] An updating module is used to update the parameters of the image encoder according to the difference between the predicted value and the true value sample image corresponding to the label and the restored image, until the updated image encoder meets the training requirements, thereby obtaining the pre-trained image encoder.

[0140] In some embodiments, the label of the training sample further includes a ground position and a parallel line position, the shooting parameter at least includes the shooting angle, and the image generating device further includes:

[0141] The local loss value calculation module is used to calculate the local loss value for each of the above labels of the above training samples according to the above true value corresponding to the above label, the above predicted value and the loss function corresponding to the above label.

[0142] The global loss value calculation module is used to determine the global loss value based on the above local loss values.

[0143] A parameter updating module is used to update the parameters of the image encoder according to the global loss value until the updated image encoder converges to obtain the pre-trained image encoder.

[0144] In some embodiments, the target reference image acquisition module 41 includes:

[0145] The shooting parameter requirement determining unit is used to determine the shooting parameter requirement according to the constraint condition when the constraint condition does not include the target reference image, and the shooting parameter requirement at least includes the shooting angle requirement.

[0146] The target reference image determination unit is used to determine the target reference image according to the shooting parameter requirements.

[0147] In some embodiments, the target reference image acquisition module 41 further includes:

[0148] The candidate reference image determination unit is used to determine the candidate reference image that meets the above shooting parameter requirements according to the above shooting parameter requirements.

[0149] A screening unit is used to determine the target reference image from each of the candidate reference images according to the degree of matching between the candidate reference images and the shooting parameter requirements when the number of the candidate reference images is greater than 1, or to determine the target reference image according to a selection instruction, wherein the selection instruction is used to indicate the candidate reference image selected from each of the candidate reference images.

[0150] In some embodiments, the target reference image acquisition module 41 further includes:

[0151] The candidate reference image generating unit is used to generate the candidate reference image according to the shooting parameter requirements when the accuracy of the shooting parameter requirements meets the precision requirements.

[0152] The candidate reference image screening unit is used to screen the images that meet the above shooting parameter requirements from the pre-stored images to obtain the above candidate reference images when the above accuracy required by the above shooting parameters does not meet the above precision requirements.

[0153] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0154] Embodiment three:

[0155] Figure 5 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 5 As shown, the electronic device 5 of this embodiment includes: at least one processor 50 ( Figure 5Only one processor is shown in the figure), a memory 51, and a computer program 52 stored in the memory 51 and executable on the at least one processor 50, wherein the processor 50 implements the steps in any of the above-mentioned method embodiments when executing the computer program 52.

[0156] The electronic device 5 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will appreciate that Figure 5 It is only an example of the electronic device 5 and does not constitute a limitation on the electronic device 5. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0157] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0158] In some embodiments, the memory 51 may be an internal storage unit of the electronic device 5, such as a hard disk or memory of the electronic device 5. In other embodiments, the memory 51 may also be an external storage device of the electronic device 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. Further, the memory 51 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 51 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 51 may also be used to temporarily store data that has been output or is to be output.

[0159] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0160] An embodiment of the present application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the above-mentioned method embodiments when executing the computer program.

[0161] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0162] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0163] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0164] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0165] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0166] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0167] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0168] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image generation method, characterized in that: include: Determine a target reference image according to the constraint condition, wherein the target reference image at least reflects the shooting angle requirement; Encoding the target reference image using a pre-trained image encoder to obtain an image code; Based on the constraint conditions and the image coding, the noise image is subjected to inverse diffusion processing to obtain a target image.

2. The image generation method according to claim 1, characterized in that: Before encoding the target reference image by the pre-trained image encoder to obtain the image encoding, the method further includes: Acquire a plurality of training samples with different shooting angles, wherein labels of the training samples at least include shooting parameters, and the shooting parameters at least include the shooting angle; The training sample is encoded by the constructed image encoder to obtain a sample code; Decoding the sample code to obtain a predicted value corresponding to the label; The parameters of the image encoder are updated according to the difference between the predicted value and the true value corresponding to the label until the updated image encoder meets the training requirements, thereby obtaining the pre-trained image encoder.

3. The image generation method according to claim 2, characterized in that: The label also includes a ground position and a parallel line position, and the parameters of the image encoder are updated according to the difference between the predicted value and the true value corresponding to the label until the updated image encoder meets the training requirements to obtain the pre-trained image encoder, including: For each of the labels of the training samples, a local loss value is calculated according to the true value corresponding to the label, the predicted value, and the loss function corresponding to the label; Determine a global loss value according to each of the local loss values; The parameters of the image encoder are updated according to the global loss value until the updated image encoder converges to obtain the pre-trained image encoder.

4. The image generation method according to claim 1, wherein: The step of determining the target reference image according to the constraint conditions comprises: In a case where the constraint condition does not include the target reference image, determining a shooting parameter requirement according to the constraint condition, the shooting parameter requirement at least including the shooting angle requirement; The target reference image is determined according to the shooting parameter requirement.

5. The image generation method according to claim 4, characterized in that: The step of determining the target reference image according to the shooting parameter requirement includes: Determining, according to the shooting parameter requirement, a candidate reference image that meets the shooting parameter requirement; When the number of the candidate reference images is greater than 1, the target reference image is determined from each of the candidate reference images according to the degree of matching between the candidate reference images and the shooting parameter requirements, or the target reference image is determined according to a selection instruction, and the selection instruction is used to indicate the candidate reference image selected from each of the candidate reference images.

6. The image generation method according to claim 5, characterized in that: The determining, according to the shooting parameter requirement, a candidate reference image that meets the shooting parameter requirement includes: In a case where the accuracy required by the shooting parameter meets the precision requirement, generating the candidate reference image according to the shooting parameter requirement; In the case that the accuracy required by the shooting parameters does not meet the precision requirement, the image that meets the shooting parameter requirement is screened out from various pre-stored images to obtain the candidate reference image.

7. An image generating device, characterized in that: include: A target reference image acquisition module, used to determine a target reference image according to a constraint condition, wherein the target reference image at least reflects a shooting angle requirement; An encoding module, used for encoding the target reference image through a pre-trained image encoder to obtain an image code; A generating module is used to perform inverse diffusion processing on the noise image based on the constraint condition and the image coding to obtain a target image.

8. The image generating device according to claim 7, wherein: Also includes: Acquire a plurality of training samples with different shooting angles, wherein labels of the training samples at least include shooting parameters, and the shooting parameters at least include the shooting angle; A sample encoding module, used for encoding the training sample through a constructed image encoder to obtain a sample code; A decoding module, used to decode the sample code to obtain a predicted value corresponding to the label; An updating module is used to update the parameters of the image encoder according to the difference between the predicted value and the true value corresponding to the label, until the updated image encoder meets the training requirements, thereby obtaining the pre-trained image encoder.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images