Image generation method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to an image generation method and device, electronic equipment and storage medium, the method comprising: determining a target parameter corresponding to a target object in an original image based on the original image using a parameter prediction network; configuring parameters in a feature extraction network using the target parameter; extracting features of the target object from the original image using the configured feature extraction network; and performing noise reduction processing on a preset noise image based on the features of the target object in the original image using a noise reduction network to obtain a target image; the target image comprising a derivative object; and the exclusive features of the derivative object being consistent with the exclusive features of the target object. The technical solution provided by the present disclosure does not require training of the feature extraction network, does not have the problem of long training time, can reduce the performance requirements of the equipment, and can promote the application and promotion of the exclusive feature preservation technology.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to an image generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, especially the application of deep learning in image processing, image generation technology based on guided maps has been realized. Among the many application scenarios of image generation technology, portrait preservation is a very important one. Portrait preservation aims to ensure that the identity features and facial details of a specific person are accurately and consistently preserved during the image generation process, making the generated image highly similar to the original person, and even to a certain extent, indistinguishable from the original person.

[0003] However, in practical applications, existing technical solutions typically require users to provide multiple portrait images of the same person. These image data are then used to train a model to construct a portrait-preserving model specific to that person. This model can then be used to generate images of that person. This training process is usually time-consuming and places high demands on device performance, severely limiting the application and promotion of portrait-preserving technology. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides an image generation method, apparatus, electronic device, and storage medium.

[0005] In a first aspect, this disclosure provides an image generation method. The image generation model used to implement the image generation method includes a parameter prediction network, a feature extraction network, and a denoising network. The parameter prediction network and the denoising network are both connected to the feature extraction network. The method includes:

[0006] Using the parameter prediction network, target parameters corresponding to the target object in the original image are determined based on the original image;

[0007] The parameters in the feature extraction network are configured using the target parameters;

[0008] Using the configured feature extraction network, features of the target object are extracted from the original image;

[0009] Using the denoising network, a preset noisy image is denoised based on the features of the target object in the original image to obtain a target image; the target image includes a derived object; the exclusive features of the derived object are consistent with the exclusive features of the target object.

[0010] Secondly, this disclosure also provides an image generation apparatus, wherein the image generation model used to implement the image generation method includes a parameter prediction network, a feature extraction network, and a denoising network, wherein the parameter prediction network and the denoising network are both connected to the feature extraction network, and the apparatus includes:

[0011] A parameter prediction module is used to determine, based on the original image, target parameters corresponding to a target object in the original image using the parameter prediction network; the original image includes the target object.

[0012] A configuration module is used to configure the parameters in the feature extraction network using the target parameters;

[0013] The feature extraction module is used to extract features of the target object from the original image using the configured feature extraction network;

[0014] An image generation module is used to perform noise reduction processing on a preset noisy image based on the features of the target object in the original image using the noise reduction network to obtain a target image; the target image includes a derived object; the exclusive features of the derived object are consistent with the exclusive features of the target object.

[0015] Thirdly, this disclosure also provides an electronic device, the electronic device comprising:

[0016] One or more processors;

[0017] Storage device for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the image generation method as described above.

[0019] Fourthly, this disclosure also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the image generation method described above.

[0020] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0021] The technical solution provided in this disclosure involves setting up a parameter prediction network to determine target parameters corresponding to a target object in the original image based on the original image; configuring the parameters in a feature extraction network using the target parameters; extracting features of the target object from the original image using the configured feature extraction network; and using a denoising network to denoise a preset noisy image based on the features of the target object in the original image to obtain the target image. The target image includes a derived object, and the exclusive features of the derived object are consistent with the exclusive features of the target object. Essentially, when the derived object in the generated image needs to maintain the exclusive features of the target object in the original image, a dedicated feature extraction network is not trained for the target object. Instead, parameters matching the target object are directly predicted for configuring the feature extraction network, and these parameters are used to configure the feature extraction network. Since the technical solution provided in this application does not require training the feature extraction network, there is no problem of long training time, and the performance requirements of the device are reduced, which can promote the application and popularization of exclusive feature preservation technology. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0023] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram of an image generation model provided in an embodiment of this disclosure;

[0025] Figure 2 A flowchart of an image generation method provided in this disclosure embodiment;

[0026] Figure 3 A schematic diagram of another image generation model provided in an embodiment of this disclosure;

[0027] Figure 4 A flowchart of another image generation method provided in this disclosure embodiment;

[0028] Figure 5 This is a schematic diagram of the structure of an image generation device according to an embodiment of the present disclosure;

[0029] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0030] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0031] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0032] Figure 1 This is a schematic diagram of an image generation model provided in an embodiment of this disclosure. Figure 2 This is a flowchart illustrating an image generation method provided in this embodiment. This embodiment is applicable to situations where image generation is performed on a client side. The method can be executed by an image generation device, which can be implemented in software and / or hardware. This device can be configured in an electronic device, such as a terminal, including but not limited to smartphones, PDAs, tablets, wearable devices with displays, desktop computers, laptops, all-in-one computers, smart home devices, etc. Alternatively, this embodiment can be applied to situations where image generation is performed on a server side. The method can be executed by an image generation device, which can be implemented in software and / or hardware. This device can be configured in an electronic device, such as a server.

[0033] See Figure 1 This image generation model can implement the image generation method provided in this application. The image generation model includes a parameter prediction network, a feature extraction network, and a denoising network. Both the parameter prediction network and the denoising network are connected to the feature extraction network.

[0034] Figure 2 The method may specifically include:

[0035] S110. Using a parameter prediction network, determine the target parameters corresponding to the target object in the original image based on the original image.

[0036] The original image can be, for example, an image provided to the image generation model during its use to guide the model in image generation. The original image includes the target object, which is the object in the original image that needs to be highlighted or emphasized; it is the object whose unique features need to be preserved in the target image (i.e., the image to be generated). This application does not limit the specific things that the target object refers to. In some scenarios, the target object can be a person, animal, or other object.

[0037] The specific characteristics of a target object can refer to one or more specific features or attributes of the target object that can uniquely distinguish it from other objects. These features or attributes are unique to the target object and can be clearly distinguished even among similar objects. For example, if the target object is a face, the specific characteristic information of the target object may include the color and shape of the eyes, the shape of the nose, scars or moles, etc. If the target object is an animal, the specific characteristic information of the target object may include fur color, spots, stripes, and body shape, etc. If the target object is an object, the specific characteristic information of the target object may include shape, size, material, and surface texture, etc.

[0038] When a feature extraction network is configured with different parameters, it will focus on different image regions or feature types during the feature extraction process. For example, if the object is a face, with some parameter configurations, the feature extraction network may focus on extracting the geometric features of the face; while with other parameter configurations, the feature extraction network may focus on extracting the skin texture features of the face.

[0039] In practice, different objects often exhibit significant individual differences, thus requiring a feature extraction network adapted to them for feature extraction. The role of the parameter prediction network is to predict which parameters should be configured for the feature extraction network to preserve the features of the target object in the target image, so that the extracted features can fully reflect the characteristics of the target object.

[0040] S120. Configure the parameters in the feature extraction network using the target parameters.

[0041] S130. Using the configured feature extraction network, extract the features of the target object from the original image.

[0042] Since the target parameters used by the configuration feature extraction network correspond to the target object, this means that the configured feature extraction network is very suitable for extracting features of the target object in the original image.

[0043] S140. Using a noise reduction network, based on the features of the target object in the original image, the preset noisy image is denoised to obtain the target image; the target image includes derived objects; the exclusive features of the derived objects are consistent with the exclusive features of the target objects.

[0044] The preset noise image can be, for example, a pre-specified noise image.

[0045] The essence of this step is to use the features of the target object in the original image to influence the noise reduction process of the preset noisy image, thereby making the derived object in the target image highly similar to the target object in the original image. To a certain extent, the derived object in the target image and the target object in the original image can be regarded as the same object. In other words, the derived object retains the unique features of the target object.

[0046] The above technical solution involves setting up a parameter prediction network to determine target parameters corresponding to the target object in the original image; configuring the parameters in the feature extraction network using the target parameters; extracting the features of the target object from the original image using the configured feature extraction network; and using a denoising network to denoise a preset noisy image based on the features of the target object in the original image to obtain the target image. The target image includes derived objects, and the exclusive features of the derived objects are consistent with the exclusive features of the target objects. Essentially, when the derived objects in the generated image need to maintain the exclusive features of the target objects in the original image, a dedicated feature extraction network is not trained for the target objects. Instead, the parameters matching the target objects are directly predicted for configuring the feature extraction network, and these parameters are used to configure the feature extraction network. Since the technical solution provided in this application does not require training the feature extraction network, there is no problem of long training time, and the performance requirements of the equipment are reduced, which can promote the application and popularization of exclusive feature preservation technology (especially portrait preservation technology).

[0047] It should be noted that, in practice, the features of the target object extracted from the original image may include both object-specific features and general features.

[0048] Furthermore, based on the above technical solutions, Figure 3 This is a schematic diagram of another image generation model provided in an embodiment of this disclosure. Figure 4 A flowchart illustrating another image generation method provided in this disclosure. See also... Figure 3 In this image generation model, the parameter prediction network includes a first parameter prediction network and a second parameter prediction network; the feature extraction network includes a dedicated feature extraction network and a general feature extraction network; the first parameter prediction network is connected to the dedicated feature extraction network; the second parameter prediction network is connected to the general feature extraction network; and the dedicated feature extraction network and the general feature extraction network are connected to the noise reduction network.

[0049] S210. Using a first parameter prediction network, determine a first target parameter corresponding to the target object in the original image based on the original image; using a second parameter prediction network, determine a second target parameter corresponding to the target object in the original image based on the original image.

[0050] S220. Configure the parameters in the dedicated feature extraction network using the first target parameter; configure the parameters in the general feature extraction network using the second target parameter.

[0051] S230. Using the configured dedicated feature extraction network, extract the dedicated features of the target object from the original image; using the configured general feature extraction network, extract the general features of the target object from the original image.

[0052] S240. Using a noise reduction network, based on the specific and general features of the target object in the original image, the preset noisy image is denoised to obtain the target image.

[0053] Specific features are key factors used to distinguish different objects or individuals. General features, on the other hand, are characteristics shared by different objects or individuals. Taking a person as an example, by extracting specific features, we can ensure that the target image retains the person's identity characteristics after noise reduction, allowing the portrait to be accurately identified even after various processing steps. General features, however, are shared by most objects of the same type (such as a crowd), such as the basic facial structure and symmetry.

[0054] General features help shape the overall image of the target object, while specific features are mainly used to improve the recognizability of the target object. The essence of this technical solution is that when extracting features from the target object, both specific and general features are extracted simultaneously, so that the generated derived object is reasonable and recognizable, thereby achieving the goal of treating the derived object and the target object as the same object.

[0055] By setting up a prediction network with first parameters, a first target parameter corresponding to the target object in the original image is determined based on the original image; a second target parameter is determined based on the original image, and a second target parameter is determined based on the original image; the parameters in the specific feature extraction network are configured using the first target parameter; the parameters in the general feature extraction network are configured using the second target parameter; the specific feature extraction network is used to extract the specific features of the target object from the original image; and the general feature extraction network is used to extract the general features of the target object from the original image. Essentially, this process extracts and processes the general and specific features of the original image separately, which helps to highlight the specific features during the noise reduction process and avoids the specific features from being mixed with the general features and being masked by the general features, thus preventing the derived object from having low recognition in the target image.

[0056] Based on the above technical solution, optionally, the image generation model further includes a first encoder; the first encoder is connected to a first parameter prediction network; S210 may include: encoding the original image using the first encoder to obtain a first encoding result of the original image; using the first parameter prediction network, based on the first encoding result of the original image, determining a first target parameter corresponding to the original image and a first feature of the target object; S230 may include: extracting the target object's specific features from the first feature of the target object using a configured specific feature extraction network. The first feature of the target object is the image feature related to the target object in the first encoding result of the original image. Essentially, this setup means that before inputting the original image into the first parameter prediction network, the first encoder encodes the original image, and the first feature of the target object is passed to the specific feature extraction network via the first parameter prediction network. In practice, the first parameter prediction network and the specific feature extraction network often cannot directly process the original image and need to encode it. The advantage of this setup is that only one first encoder is used, instead of configuring separate encoders for the first parameter prediction network and the specific feature extraction network. This setup can simultaneously meet the image processing needs of both the first parameter prediction network and the specific feature extraction network, simplifying the architecture of the image generation model.

[0057] Similarly, the image generation model also includes a second encoder; the second encoder is connected to a second parameter prediction network; S210 may include: encoding the original image using the second encoder to obtain a second encoding result of the original image; using the second parameter prediction network, based on the second encoding result of the original image, determining the second target parameter corresponding to the original image and the second feature of the target object; S230 may include: using a configured general feature extraction network to extract the general feature of the target object from the second feature of the target object. The second feature of the target object is the image feature related to the target object in the second encoding result of the original image. Before inputting the original image into the second parameter prediction network, the original image is encoded using the second encoder, and the second feature of the target object is passed to the general feature extraction network through the second parameter prediction network. The advantage of this setup is that only one second encoder is set up, instead of configuring separate encoders for the second parameter prediction network and the general feature extraction network, which can simultaneously meet the image processing needs of the second parameter prediction network and the general feature extraction network, simplifying the architecture of the image generation model.

[0058] Based on the above technical solutions, optionally, the training process of the image generation model includes a first stage and a second stage, and the method for training the image generation model includes: in the first stage, training a second parameter prediction network to enable the second parameter prediction network to predict parameters applicable to a general feature extraction network; in the second stage, training a first parameter prediction network to enable the first parameter prediction network to predict parameters applicable to a specific feature extraction network.

[0059] Further, in the first stage, the second parameter prediction network is trained to enable it to predict parameters applicable to a general feature extraction network. This includes: acquiring sample image pairs, which include a first sample image and a second sample image; the difference between the first sample image and the second sample image is that the first sample image lacks a target object, while the second sample image includes the target object; and in the first stage, the second parameter prediction network is trained based on the sample image pairs to enable it to predict parameters applicable to a general feature extraction network.

[0060] For example, if an image generation model is trained and it is desired to have a human face preservation function, the target object can be set as a face, the second sample image is an image including a complete human face, and the first sample image is an image after the human face in the second sample image is occluded.

[0061] Further, in the second stage, the first parameter prediction network is trained to enable it to predict parameters suitable for a dedicated feature extraction network. This includes: acquiring a third sample image and a fourth sample image in the second stage; both the third and fourth sample images include sample objects; obtaining sample text corresponding to the third sample image based on the third sample image; the sample text describing the third sample image; training the first parameter prediction network based on the third sample image and the sample text to determine the parameters in the text-related cross-attention module of the first parameter prediction network; keeping the parameters in the text-related cross-attention module of the first parameter prediction network fixed, training the first parameter prediction network based on the fourth sample image to determine the parameters in the image-related cross-attention module of the first parameter prediction network.

[0062] The sample object pair refers to the objects that need to be preserved, which are used to simulate the target objects during the image generation mode usage phase. The sample objects in the third sample image and the sample objects in the fourth sample image can be the same sample objects or different sample objects.

[0063] The first parameter prediction network includes two cross-attention modules. One cross-attention module is text-related, used to interpret the text's description of the features of the object to be preserved and translate it into a guide for image feature extraction. The other cross-attention module is image-related, used to filter image features and capture local correlations to supplement details.

[0064] It's important to note that training in either the first or second stage is not performed on a single network (such as the first or second parameter prediction network), but rather on the entire image generation model. During training, a loss function is determined in the image domain; that is, the parameters of the first or second parameter prediction network are adjusted by comparing the differences between the objects in the predicted image output by the image generation model and the objects in the input image. While adjusting the parameters, the parameters of the denoising model remain frozen.

[0065] The above technical solution, by setting up separate training for the first parameter prediction network and the second parameter prediction network, facilitates the rapid convergence of the image generation model and reduces the training time of the image generation model.

[0066] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0067] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0068] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0069] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0070] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0071] Figure 5 This is a schematic diagram of an image generation device according to an embodiment of this disclosure. The image generation device provided in this embodiment can be configured in a client or a server. The image generation model used to implement the image generation device includes a parameter prediction network, a feature extraction network, and a denoising network, wherein the parameter prediction network and the denoising network are both connected to the feature extraction network. See also... Figure 5 The image generation device specifically includes:

[0072] The parameter prediction module 310 is used to determine, based on the original image, target parameters corresponding to the target object in the original image using the parameter prediction network; the original image includes the target object.

[0073] Configuration module 320 is used to configure the parameters in the feature extraction network using the target parameters;

[0074] Feature extraction module 330 is used to extract features of the target object from the original image using the configured feature extraction network;

[0075] The image generation module 340 is used to perform noise reduction processing on a preset noisy image based on the features of the target object in the original image using the noise reduction network to obtain a target image; the target image includes a derived object; the exclusive features of the derived object are consistent with the exclusive features of the target object.

[0076] Furthermore, the parameter prediction network includes a first parameter prediction network and a second parameter prediction network; the feature extraction network includes a dedicated feature extraction network and a general feature extraction network; the first parameter prediction network is connected to the dedicated feature extraction network; the second parameter prediction network is connected to the general feature extraction network; the dedicated feature extraction network and the general feature extraction network are also connected to the noise reduction network.

[0077] The parameter prediction module 310 is configured to: use the first parameter prediction network to determine a first target parameter corresponding to a target object in the original image based on the original image; and use the second parameter prediction network to determine a second target parameter corresponding to a target object in the original image based on the original image.

[0078] The configuration module 320 is used to: configure the parameters in the dedicated feature extraction network using the first target parameter; and configure the parameters in the general feature extraction network using the second target parameter.

[0079] The feature extraction module 330 is used to: extract the specific features of the target object from the original image using the configured dedicated feature extraction network; and extract the general features of the target object from the original image using the configured general feature extraction network.

[0080] The image generation module 340 is used to: use a noise reduction network to perform noise reduction processing on a preset noisy image based on the specific features and general features of the target object in the original image, so as to obtain the target image.

[0081] Furthermore, the image generation model also includes a first encoder; the first encoder is connected to the first parameter prediction network;

[0082] The parameter prediction module 310 is used for:

[0083] The original image is encoded using the first encoder to obtain a first encoding result of the original image;

[0084] Using the first parameter prediction network, based on the first encoding result of the original image, determine the first target parameter corresponding to the original image and the first feature of the target object;

[0085] The feature extraction module 330 is used for:

[0086] The configured dedicated feature extraction network is used to extract the dedicated features of the target object from the first feature of the target object.

[0087] Furthermore, the image generation model also includes a second encoder; the second encoder is connected to the second parameter prediction network;

[0088] The parameter prediction module 310 is used for:

[0089] The original image is encoded using the second encoder to obtain a second encoding result of the original image;

[0090] Using the second parameter prediction network, based on the second encoding result of the original image, the second target parameter corresponding to the original image and the second feature of the target object are determined;

[0091] The feature extraction module 330 is used for:

[0092] The configured general feature extraction network is used to extract the general features of the target object from the second feature of the target object.

[0093] Furthermore, the training process of the image generation model includes a first stage and a second stage, and the device further includes a training module for:

[0094] In the first stage, the second parameter prediction network is trained so that the second parameter prediction network has the ability to predict parameters applicable to general feature extraction networks;

[0095] In the second stage, the first parameter prediction network is trained to enable it to predict parameters suitable for a dedicated feature extraction network.

[0096] Furthermore, the training module is used for:

[0097] In the first stage, training the second parameter prediction network to enable it to predict parameters applicable to general feature extraction networks includes:

[0098] Obtain a pair of sample images, the pair of sample images including a first sample image and a second sample image; the difference between the first sample image and the second sample image is that the first sample image lacks a sample target object, while the second sample image includes a sample target object.

[0099] In the first stage, the second parameter prediction network is trained based on the sample image pairs so that the second parameter prediction network has the ability to predict parameters applicable to general feature extraction networks.

[0100] Furthermore, the training module is used for:

[0101] In the second stage, a third sample image and a fourth sample image are acquired; both the third sample image and the fourth sample image include sample objects.

[0102] Based on the third sample image, sample text corresponding to the third sample image is obtained; the sample text is used to describe the third sample image.

[0103] Based on the third sample image and the sample text, the first parameter prediction network is trained to determine the parameters in the text-related cross-attention module of the first parameter prediction network.

[0104] Keeping the parameters of the text-related cross-attention module in the first parameter prediction network fixed, the first parameter prediction network is trained based on the fourth sample image to determine the parameters of the image-related cross-attention module in the first parameter prediction network.

[0105] The image generation apparatus provided in this disclosure can execute the steps performed by the client or server in the image generation method provided in this disclosure, and has the execution steps and beneficial effects, which will not be described in detail here.

[0106] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 6 The diagram illustrates a structural schematic suitable for implementing the electronic device 1000 in the embodiments of this disclosure. The electronic device 1000 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0107] like Figure 6 As shown, the electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003 to implement the image generation method as described in the embodiments of this disclosure. The RAM 1003 also stores various programs and information required for the operation of the electronic device 1000. The processing device 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0108] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to exchange information with other devices wirelessly or via wired communication. Although Figure 6 An electronic device 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0109] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the image generation method as described above. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from storage device 1008, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0110] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include information signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated information signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0111] In some implementations, clients and servers may communicate using any known or future network protocol, such as HTTP (Hypertext Transfer Protocol), and may interconnect with any form or medium of digital information communication (e.g., a communication network). Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any known or future network.

[0112] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0113] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0114] Using the parameter prediction network, target parameters corresponding to the target object in the original image are determined based on the original image;

[0115] The parameters in the feature extraction network are configured using the target parameters;

[0116] The configured feature extraction network is used to extract features of the target object from the original image.

[0117] Using the denoising network, a preset noisy image is denoised based on the features of the target object in the original image to obtain a target image; the target image includes a derived object; the exclusive features of the derived object are consistent with the exclusive features of the target object.

[0118] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

[0119] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0121] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0122] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0123] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] According to one or more embodiments of this disclosure, this disclosure provides an electronic device, including:

[0125] One or more processors;

[0126] Memory, used to store one or more programs;

[0127] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the image generation methods provided in this disclosure.

[0128] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements an image generation method as described in any of the present disclosure.

[0129] This disclosure also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the image generation method described above.

[0130] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0131] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image generation method, characterized in that, The image generation model used to implement the image generation method includes a parameter prediction network, a feature extraction network, and a denoising network. Both the parameter prediction network and the denoising network are connected to the feature extraction network. The method includes: Using the parameter prediction network, target parameters corresponding to the target object in the original image are determined based on the original image; The parameters in the feature extraction network are configured using the target parameters; The configured feature extraction network is used to extract features of the target object from the original image. Using the denoising network, a preset noisy image is denoised based on the features of the target object in the original image to obtain a target image; the target image includes a derived object; the exclusive features of the derived object are consistent with the exclusive features of the target object.

2. The method according to claim 1, characterized in that, The parameter prediction network includes a first parameter prediction network and a second parameter prediction network; the feature extraction network includes a dedicated feature extraction network and a general feature extraction network; the first parameter prediction network is connected to the dedicated feature extraction network; the second parameter prediction network is connected to the general feature extraction network; the dedicated feature extraction network and the general feature extraction network are also connected to the noise reduction network; The step of using the parameter prediction network to determine the target parameters corresponding to the target object in the original image based on the original image includes: using the first parameter prediction network to determine the first target parameter corresponding to the target object in the original image based on the original image; and using the second parameter prediction network to determine the second target parameter corresponding to the target object in the original image based on the original image. The step of configuring the parameters in the feature extraction network using the target parameters includes: configuring the parameters in the dedicated feature extraction network using the first target parameters; and configuring the parameters in the general feature extraction network using the second target parameters. The step of extracting features of the target object from the original image using the configured feature extraction network includes: extracting specific features of the target object from the original image using the configured dedicated feature extraction network; and extracting general features of the target object from the original image using the configured general feature extraction network. The step of using a denoising network to denoise a preset noisy image based on the features of the target object in the original image to obtain a target image includes: using a denoising network to denoise a preset noisy image based on the specific and general features of the target object in the original image to obtain a target image.

3. The method according to claim 2, characterized in that, The image generation model further includes a first encoder; the first encoder is connected to the first parameter prediction network. The step of using the first parameter prediction network to determine the first target parameter corresponding to the original image based on the original image includes: The original image is encoded using the first encoder to obtain a first encoding result of the original image; Using the first parameter prediction network, based on the first encoding result of the original image, determine the first target parameter corresponding to the original image and the first feature of the target object; The step of extracting the specific features of the target object from the original image using the configured specific feature extraction network includes: The configured dedicated feature extraction network is used to extract the dedicated features of the target object from the first feature of the target object.

4. The method according to claim 3, characterized in that, The image generation model further includes a second encoder; the second encoder is connected to the second parameter prediction network. The step of using the second parameter prediction network to determine the second target parameter corresponding to the original image based on the original image includes: The original image is encoded using the second encoder to obtain a second encoding result of the original image; Using the second parameter prediction network, based on the second encoding result of the original image, the second target parameter corresponding to the original image and the second feature of the target object are determined; The step of extracting general features of the target object from the original image using the configured general feature extraction network includes: The configured general feature extraction network is used to extract the general features of the target object from the second feature of the target object.

5. The method according to claim 4, characterized in that, The training process of the image generation model includes a first stage and a second stage, and the method for training the image generation model includes: In the first stage, the second parameter prediction network is trained so that the second parameter prediction network has the ability to predict parameters applicable to general feature extraction networks; In the second stage, the first parameter prediction network is trained to enable it to predict parameters suitable for a dedicated feature extraction network.

6. The method according to claim 5, characterized in that, In the first stage, training the second parameter prediction network to enable it to predict parameters applicable to general feature extraction networks includes: Obtain a pair of sample images, the pair of sample images including a first sample image and a second sample image; the difference between the first sample image and the second sample image is that the first sample image lacks a sample target object, while the second sample image includes a sample target object. In the first stage, the second parameter prediction network is trained based on the sample image pairs so that the second parameter prediction network has the ability to predict parameters applicable to general feature extraction networks.

7. The method according to claim 6, characterized in that, In the second stage, the first parameter prediction network is trained to enable it to predict parameters suitable for a dedicated feature extraction network, including: Acquire a third sample image and a fourth sample image; both the third sample image and the fourth sample image include sample objects; Based on the third sample image, sample text corresponding to the third sample image is obtained; the sample text is used to describe the third sample image. Based on the third sample image and the sample text, the first parameter prediction network is trained to determine the parameters in the text-related cross-attention module of the first parameter prediction network. Keeping the parameters of the text-related cross-attention module in the first parameter prediction network fixed, the first parameter prediction network is trained based on the fourth sample image to determine the parameters of the image-related cross-attention module in the first parameter prediction network.

8. An image generation apparatus, characterized in that, The image generation model used in the image generation device includes a parameter prediction network, a feature extraction network, and a denoising network. Both the parameter prediction network and the denoising network are connected to the feature extraction network. The device includes: A parameter prediction module is used to determine, based on the original image, target parameters corresponding to a target object in the original image using the parameter prediction network; the original image includes the target object. A configuration module is used to configure the parameters in the feature extraction network using the target parameters; The feature extraction module is used to extract features of the target object from the original image using the configured feature extraction network; An image generation module is used to perform noise reduction processing on a preset noisy image based on the features of the target object in the original image using the noise reduction network to obtain a target image; the target image includes a derived object; the exclusive features of the derived object are consistent with the exclusive features of the target object.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images