Image generating method, device and electronic device

Abstract

The present disclosure provides an image generation method, apparatus, and electronic device, the image generation method including: obtaining a target text, a sketch image, and a noise image; obtaining a fusion vector based on the target text and the sketch image; and performing a denoising process on the noise image based on the fusion vector to obtain a target image that matches the target text and the sketch image. The embodiments of the present disclosure improve the effect of image generation and achieve the purpose of generating images according to a user's intention.

Description

[Technical Field]

[0001] This application claims priority from Chinese Patent Application No. 202211634290.6 filed on December 19, 2022, and the entire contents of the above-mentioned Chinese patent application are hereby incorporated by reference into this application.

[0002] The present disclosure relates to image generation methods, devices and electronic equipment. [Background technology]

[0003] With the application and development of artificial intelligence technology in the field of imaging, image generation technology has emerged. Currently, there is a demand for purposeful image generation according to user needs. With the continuous development of multimodal technology, text-based image generation technology has made rapid progress and achieved excellent results. However, image description based on linguistic text has limitations, making it difficult to meet user needs. Summary of the Invention

[0004] The present disclosure provides image generation methods, devices and electronic equipment.

[0005] In a first aspect, the present disclosure provides a method of generating an image, comprising: obtaining a target text, a sketch image and a noise image; obtaining a fusion vector based on the target text and the sketch image; and performing a denoising process on the noise image based on the fusion vector to obtain a target image that matches the target text and the sketch image.

[0006] In a second aspect, the present disclosure provides an image generation device, comprising: an acquisition module for acquiring a target text, a sketch image, and a noise image; a fusion module for obtaining a fusion vector based on the target text and the sketch image; and a denoising module for performing a denoising process on the noise image based on the fusion vector to obtain a target image that matches the target text and the sketch image.

[0007] In a third aspect, the present disclosure provides a computer-readable storage medium having stored thereon a computer program that, when executed by a processor, implements the method of any one of the first aspects above.

[0008] In a fourth aspect, the present disclosure provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program implementing the method of any one of the first aspect when the processor executes the program.

[0009] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. [Brief explanation of the drawings]

[0010] In order to more clearly describe the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings that need to be used in the description of the embodiments. Obviously, the drawings in the following description are merely some embodiments described in the present disclosure, and those skilled in the art can further obtain other drawings based on these drawings without any creative work.

[0011] [Figure 1] FIG. 1 is a schematic diagram of a scene for image generation according to an exemplary embodiment of the present disclosure. [Figure 2] 1 is a flowchart of an image generation method according to an exemplary embodiment of the present disclosure. [Figure 3] FIG. 2 is a schematic diagram of an example of image generation according to an exemplary embodiment of the present disclosure. [Figure 4] FIG. 1 is a block diagram of an image production device according to an exemplary embodiment of the present disclosure. [Figure 5] FIG. 1 is a schematic block diagram of an electronic device according to some embodiments of the present disclosure. [Figure 6] FIG. 1 is a schematic block diagram of another electronic device according to some embodiments of the present disclosure. [Figure 7] 1 is a schematic diagram of a storage medium according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] In order to allow those skilled in the art to better understand the technical solutions in the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the present disclosure, and it is obvious that the described embodiments are only some embodiments of the present disclosure, and not all embodiments, and all other embodiments obtained by those skilled in the art based on the embodiments in the present disclosure without making creative efforts should fall within the protection scope of the present disclosure.

[0013] When the following description refers to the drawings, unless otherwise stated, the same numerals in different drawings represent the same or similar elements. The embodiments described in the following illustrative examples do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0014] The terms used in this disclosure are for the purpose of describing particular embodiments only and are not intended to limit the disclosure. As used in this disclosure, the singular forms "a," "the," and "the" are intended to include many unless the context clearly indicates otherwise. The term "and / or," as used herein, refers to and should be understood to include any or all possible combinations of one or more associated listed items.

[0015] While terms such as "first," "second," "third," etc. may be used to describe various pieces of information in this disclosure, it should be understood that such information should not be limited to these terms. These terms are used only to distinguish between the same types of information. For example, first information could be referred to as second information, and similarly, second information could be referred to as first information, without departing from the scope of this disclosure. Depending on the context, the term "if" as used herein can be interpreted as "when," "when," or "in response to determining."

[0016] With the application and development of artificial intelligence technology in the field of imaging, image generation technology has emerged. Currently, there is a great demand for purposefully generating images according to user needs. In related technologies, with the continuous development of multimodal technology, text-based image generation technology has made rapid progress and achieved excellent results. However, image descriptions based on linguistic text have limitations, making it difficult to meet user needs. For example, the image to be generated contains complex target objects (e.g., people or objects) with different shapes, positions, or postures, which are difficult to describe in text. Specifically, for example, if you want to generate an image of a dancing girl, there are many different dance poses. It is difficult for non-professional dancers to express the desired dance pose in words, making it difficult to generate an ideal image.

[0017] The present disclosure provides an image generation method that uses text information or a simple outline sketch entered by a user as a condition for image generation to guide image generation and obtain an image that meets the user's needs, thereby achieving the objective of improving the image generation effect and making the generated image match the user's needs, thereby enabling the image to be generated according to the user's intention.

[0018] As shown in Figure 1, there is a schematic diagram of an image generation scene according to an exemplary embodiment. Hereinafter, with reference to Figure 1, aspects of the present disclosure will be generally described in relation to one complete and specific application example, which describes a specific image generation process.

[0019] As shown in FIG. 1, first, a user can provide text T and a sketch image P as conditions for guiding image generation. Here, the text T may be simple descriptive text for the target image to be generated by the user, and the sketch image P may be the outline and position of a target object in the target image to be generated, which the user has simply drawn as needed. For example, if a user wants to generate an image of a dancing girl, the text T may include "dancing girl," and the sketch image P may be the outline and dancing pose of the girl, which the user has simply drawn.

[0020] Then, the text T is input to the model M1 to perform text feature extraction, thereby obtaining a text vector Rt corresponding to the text T. Here, the text vector Rt may be, for example, a text embedding vector, and the model M1 may be a model capable of extracting a text embedding vector from text; the specific type of the model M1 is not limited in this embodiment. At the same time, the sketch image P is input to the encoder E to perform image feature extraction, thereby obtaining an image vector Rp corresponding to the sketch image P. Here, the image vector Rt may be an image feature vector capable of information fusion with the text vector, and the encoder E may be an encoder capable of extracting the image feature vector from an image; the specific type of the encoder E is not limited in this embodiment.

[0021] Next, the text vector Rt and the image vector Rp are input to the model M2 to perform information fusion between the text vector Rt and the image vector Rp, thereby obtaining a fusion vector Rr that fuses information from the text T and information from the sketch image P. As can be understood, the model M2 may be any model that can perform information fusion between a vector obtained based on text and a vector obtained based on an image, and the present embodiment does not limit the specific type of the model M2. Optionally, the model M2 may be a deep learning model that uses an attention mechanism, which can better perform information fusion between the text vector Rt and the image vector Rp.

[0022] Also, a noise image S, which may be a randomly generated white Gaussian noise image, is obtained. The fusion vector Rr and the noise image S are input into a model M3, which may be a pre-trained diffusion model. The model M3 performs a one-step denoising operation on the noise image S based on the fusion vector Rr to obtain an intermediate image S1. Since the intermediate image S1 is an image obtained by removing a small portion of the noise from the noise image S, the intermediate image S1 still contains a large amount of noise. The intermediate image S1 and the fusion vector Rr are input again into the model M3, which performs a one-step denoising operation on the intermediate image S1 based on the fusion vector Rr to obtain an intermediate image S2. The subsequent similar process can be inferred in this way and will not be repeated here.

[0023] Alternatively, the text vector Rt can be input alone into the model M2, and after processing by the model M2, a vector Rc containing only the information of the text T can be obtained. After performing n steps of noise reduction to obtain an intermediate image Sn, the intermediate image Sn can be input together with the vector Rc into the model M3, which then performs one step of noise reduction on the intermediate image Sn based on the vector Rc to obtain an intermediate image Sn+1. After that, multiple similar steps of noise reduction are performed to obtain a target image. The target image is a clear image that meets the user's needs.

[0024] The encoder E, the model M1, the model M2, and the model M3 may be trained together. Specifically, a sample image may be acquired first, and then a sketch image to be mapped to the sample image may be acquired. For example, the sketch image to be mapped to the sample image may be manually drawn. As another example, a predetermined image processing algorithm may be used to extract the sketch image to be mapped to the sample image. As another example, a machine learning model may be used to extract the sketch image to be mapped to the sample image. It should be understood that any other appropriate method may be used to extract the sketch image to be mapped to the sample image, and this embodiment is not limited in this respect. The text content to be mapped to the sample image may be acquired, and for example, the text content to be mapped to the sample image may be manually edited, or the text content to be mapped to the sample image may be extracted by a machine learning model.

[0025] Next, a multi-step noise addition operation is performed on the sample image to obtain a sample noise image, and each step of the noise addition operation adds a small amount of white Gaussian noise to the sample image.The sample noise image, the text content and the sketch image mapped to the sample image are input into the corresponding models to be trained according to the information flow shown in Figure 1, and the prediction loss is calculated based on the output result.The model parameters of the model to be trained are adjusted based on the prediction loss, thereby completing the model training.

[0026] The present disclosure will now be described in detail with reference to specific examples.

[0027] 2 is a flowchart illustrating an image generation method according to an exemplary embodiment. The method may be implemented as any device, platform, server, or device cluster having computing, processing capabilities. The method may include the following steps:

[0028] As shown in FIG. 2, in step 201, a target text, a sketch image and a noise image are obtained.

[0029] In this embodiment, the noise image may be randomly generated using white Gaussian noise, and the specific method for generating the noise image is not limited in this embodiment. The target text may be a simple description of the target image to be generated by the user, and the sketch image may be the outline and position of the target object in the target image to be generated, which the user has simply drawn as needed. For example, if the user wants to generate an image of a cat, the target text may include "lazy cat sleeping," and the sketch image may be the outline of the cat in a lying position, which the user has simply drawn. Here, the target text may be obtained by editing by the user, or may be text content selected by the user from multiple candidate text options. The sketch image may be manually drawn by the user, or may be a sketch image selected by the user from multiple candidate sketch image options. Note that the target text and sketch image can be obtained by any appropriate method, and the specific method for obtaining the target text and sketch image is not limited in this embodiment.

[0030] In step 202, a fusion vector is obtained based on the target text and the sketch image.

[0031] In this example, a fusion vector can be obtained based on the target text and the sketch image. In one embodiment, a machine learning model is used to first convert the target text into a reference image, and then image features of the reference image and the sketch image are extracted, and the image features of the reference image and the sketch image are fused to obtain a fusion vector.

[0032] In another embodiment, a text vector corresponding to the target text can be obtained as a first feature vector, and an image vector corresponding to the sketch image can be obtained as a second feature vector. The first and second feature vectors can then be fused to obtain a fused vector. Optionally, the first and second feature vectors can be fused using an attention mechanism. Specifically, the target text can be converted into a text vector as the first feature vector using a language model previously trained on image-text matching data, and then the sketch image can be processed using a pre-trained encoder to obtain an image vector as the second feature vector. The first and second feature vectors can be input into a deep learning model based on an attention mechanism, and the deep learning model can fused the first and second feature vectors to obtain a fused vector. This embodiment uses an attention mechanism to fuse the target text and the sketch image, thereby improving the fusion effect.

[0033] In step 203, a denoising process is performed on the noise image based on the fusion vector to obtain a target image that matches the target text and sketch image.

[0034] In this embodiment, based on the fusion vector, a noise image is subjected to a noise reduction process according to the content described in the target text and the outline of the sketch image, thereby obtaining a target image. As shown in FIG. 3, for example, the text content of the target text 301 may include "red rose," and the sketch image 302 may be the outline of a rose simply drawn by a user. Based on the fusion vector between the target text and the sketch image, a noise reduction process is performed on the noise image to obtain a target image 303. The target image 303 matches the description in the target text 301 and the outline of the sketch image 302.

[0035] In one embodiment, a machine learning model or a preset algorithm can be used to perform one-step denoising on the noise image based on the fusion vector to obtain the target image.

[0036] In another embodiment, the noise image can be denoised using a multi-step denoising operation with a pre-trained target model, which can be a diffusion model, where the multi-step denoising operation can include at least one first operation, which is an operation that guides the denoising operation using the target text and the sketch image as conditions (i.e., an operation that performs the denoising operation based on the fusion vector).

[0037] Optionally, the multi-step noise removal operation can include at least one second operation, where the second operation is an operation that guides the noise removal process using only the target text as a condition (i.e., an operation that performs the noise removal process based on the target vector corresponding to the target text). Specifically, the inventors have found through several experiments that performing noise removal using both the target text and the sketch image as conditions is more likely to cause overfitting, that is, the sketch image is too restrictive with respect to the target image. Therefore, the inventors have conceived the idea that in the multi-step noise removal operation, in addition to using the target text and the sketch image as conditions, only the target text can be used as a condition, thereby solving the overfitting problem in the noise removal process.

[0038] As can be seen, in a multi-step denoising operation, the first and second operations may be performed alternately. Optionally, the second operation may be performed after the first operation is completed. By first performing the first operation, which is conditional on the target text and the sketch image, the sketch image can better constrain the formation of the contours of the target object in the target image, and the target text can better constrain the formation of the details in the target image, which not only avoids the problem of overfitting but also further improves the quality of the generated image.

[0039] Specifically, the first operation of any step may include the following operations: First, determine an image to be denoised, where if the first operation is the first operation of a multi-step denoising operation, the acquired noise image may be the image to be denoised; if the first operation is not the first operation, the processing result of the denoising operation of the previous step may be the image to be denoised; as can be understood, the denoising operation of the previous step may be the first operation or the second operation.

[0040] Next, a first time vector corresponding to the first operation is determined, and a target model is used to perform denoising on the image to be denoised based on the fusion vector and the first time vector. Here, the first time vector (e.g., time embedding) may be a vector that can characterise the time of the multi-step denoising operation of the first operation. For example, the first operation is the nth step in the multi-step denoising operation, and in this case, the first time vector corresponding to the first operation is a vector that can characterise the nth step. Specifically, the fusion vector and the first time vector can be merged (e.g., added) to obtain a first condition vector. Next, a target model is used to perform denoising on the image to be denoised based on the first condition vector and the fusion vector. Note that in addition to the fusion vector, other feature vectors can also be merged with the time vector together with the fusion vector to obtain a condition vector, and this embodiment is not limited in this respect.

[0041] Regarding the second operation, before performing the second operation, a text vector corresponding to the target text is obtained as a first feature vector, and a conversion process is performed on the first feature vector to obtain a target vector. The second operation of an arbitrary step may include the following operations: first, determine an image to be denoised and a second time vector corresponding to the second operation; here, if the second operation is the first operation of a multi-step denoising operation, the obtained noise image can be the image to be denoised; if the second operation is not the first operation, the processing result of the denoising operation of the previous step can be the image to be denoised. As can be understood, the denoising operation of the previous step may be the first operation or the second operation.

[0042] Next, the target model is used to denoise the image to be denoised based on the target vector and the second time vector. Specifically, the target vector and the second time vector can be merged to obtain a second condition vector. Then, the target model is used to denoise the image to be denoised based on the second condition vector and the target vector.

[0043] In this embodiment, a specific application scenario may be that when a user wants to create a work of art, he or she simply sketches the work as desired, converts the sketch into a preset image format, and then edits a simple text description. Here, the sketch includes the shape, position, and outline of the target object. Based on the sketch and the text description, the system can guide the user to generate a target image that reflects the user's thoughts.

[0044] Another specific application scenario is that during the process of solving a crime, a witness may choose a sketch from multiple candidates that most closely resembles the face and physique of a criminal suspect, or the witness may draw a sketch of the criminal suspect's face and physique themselves and then edit a text description that describes the criminal suspect's detailed characteristics. Using the sketch and text description as a guide, an image that closely resembles the criminal suspect can be generated.

[0045] This embodiment is not limited to the application scenarios described above, but can also be applied to other scenarios. The image generation method provided by the present disclosure obtains a fusion vector based on the target text and a sketch image, and then performs a noise reduction process on the noise image obtained based on the fusion vector to obtain a target image that maps to the target text and the sketch image. This embodiment combines a sketch image containing spatial information (which is difficult to describe in words) with a target text containing detailed feature descriptions (which are difficult to draw), thereby increasing the connectivity between spatial information and text information and enabling the denoised target image to meet the user's needs and better satisfy them. This improves the image generation effect and achieves the goal of generating images according to the user's intentions.

[0046] Corresponding to the above-described embodiment of the image generation method, the present disclosure further provides an embodiment of an image generation device.

[0047] As shown in FIG. 4, FIG. 4 is a block diagram of an image generating device according to an exemplary embodiment of the present disclosure, which may include an acquisition module 401, a fusion module 402, and a denoising module 403.

[0048] Here, the acquisition module 401 is used to acquire the target text, the sketch image and the noise image.

[0049] A fusion module 402 is used to obtain a fusion vector based on the target text and the sketch image.

[0050] The denoising module 403 is used to perform denoising processing on the noise image based on the fusion vector to obtain a target image that matches the target text and sketch image.

[0051] In some embodiments, the fusion module 402 may include an acquisition sub-module and a fusion sub-module (not shown).

[0052] Here, the acquisition sub-module is used to acquire a first feature vector corresponding to the target text and a second feature vector corresponding to the sketch image.

[0053] The fusion sub-module is used to perform information fusion between the first feature vector and the second feature vector to obtain a fusion vector.

[0054] In some other embodiments, the fusion submodule is configured to perform information fusion of the first feature vector and the second feature vector using an attention mechanism.

[0055] In some other embodiments, the denoising module 403 may include an execution sub-module (not shown).

[0056] Here, the execution sub-module is used to perform a multi-step denoising operation using a pre-trained target model to denoise the noisy image, where the multi-step denoising operation includes at least one first operation of performing denoising based on the fusion vector.

[0057] In some other embodiments, the execution submodule may perform any step of a first operation to determine an image to be denoised. If the first operation is the first operation of the multi-step denoising operation, the image to be denoised is the noise image. If the first operation is not the first operation, the image to be denoised is the processing result of the denoising operation of a previous step. A first time vector corresponding to the first operation is determined, and denoising the image to be denoised based on the fusion vector and the first time vector using the target model.

[0058] In another embodiment, the execution sub-module may perform the following: use a target model to denoise the image to be denoised based on the fusion vector and the first time vector, and obtain a first condition vector based on at least the fusion vector and the first time vector; use the target model to denoise the image to be denoised based on the first condition vector and the fusion vector.

[0059] In some other embodiments, the acquisition module 401 is further used to acquire a first feature vector corresponding to the target text, and perform a transformation process on the first feature vector to obtain a target vector, wherein the multi-step denoising operation includes at least one second operation of performing a denoising process based on the target vector.

[0060] In some other embodiments, the execution sub-module may perform any step of the second operation as follows: determine an image to be denoised and a second time vector corresponding to the second operation; and use a target model to denoise the image to be denoised based on the target vector and the second time vector.

[0061] In some other embodiments, the execution sub-module may first execute a first operation for a predetermined number of steps, and then execute a second operation after the first operation has been executed.

[0062] The apparatus embodiments substantially correspond to the method embodiments, so please refer to the description of some of the method embodiments for relevant parts. The above-mentioned apparatus embodiments are merely schematic, and the units shown as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules therein can be selected according to actual needs to achieve the objectives of the technical solutions of the embodiments of the present disclosure. Those skilled in the art can understand and implement them without any creative effort.

[0063] FIG. 5 is a schematic block diagram of an electronic device according to some embodiments of the present disclosure. As shown in FIG. 5, the electronic device 910 includes a processor 911 and a memory 912 and may be used to implement a client or a server. The memory 912 is used to non-temporarily store computer-executable instructions (e.g., one or more computer program modules). The processor 911 is used to execute the computer-executable instructions, which, when executed by the processor 911, can perform one or more steps of the image generation method described above and realize the image generation method described above. The memory 912 and the processor 911 may be interconnected by a bus system and / or other form of connection (not shown).

[0064] For example, the processor 911 may be a central processing unit (CPU), a graphics processing unit (GPU), or other type of processing unit having data processing capabilities and / or program execution capabilities. For example, the central processing unit (CPU) may be an X86 or ARM architecture, etc. The processor 911 may be a general-purpose processor or a special-purpose processor and may control other components in the electronic device 910 to perform desired functions.

[0065] For example, the memory 912 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile and / or nonvolatile memory. Volatile memory may include, for example, random access memory (RAM) and / or high-speed cache memory (cache). Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. The computer-readable storage medium may store one or more computer program modules, and the processor 911 may execute the one or more computer program modules to realize various functions of the electronic device 910. The computer-readable storage medium may store various applications and various data, as well as various data used and / or generated by the applications.

[0066] It should be noted that in the embodiment of the present disclosure, the specific functions and technical effects of the electronic device 910 may be referred to the description of the image generating method described above, and will not be further detailed here.

[0067] FIG. 6 is a schematic block diagram of another electronic device according to some embodiments of the present disclosure. The electronic device 920 is suitable for implementing, for example, an image generation method provided by embodiments of the present disclosure. The electronic device 920 may be a terminal device or the like and may be used to implement a client or a server. The electronic device 920 includes, but is not limited to, mobile devices such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet PCs (PADs), portable multimedia players (PMPs), in-car devices (e.g., car navigation devices), and wearable electronic devices, as well as fixed devices such as digital TVs, desktop computers, and smart home devices. Note that the electronic device 920 shown in FIG. 6 is merely an example and does not impose any limitations on the scope of functionality and use of embodiments of the present disclosure.

[0068] 6, the electronic device 920 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 921, which may perform various appropriate operations and processes in accordance with programs stored in a read-only memory (ROM) 922 or programs loaded from a storage device 928 into a random access memory (RAM) 923. The RAM 923 may also store various programs and data required for the operation of the electronic device 920. The processing unit 921, the ROM 922, and the RAM 923 are connected to one another via a bus 924. An input / output (I / O) interface 925 is also connected to the bus 924.

[0069] Typically, devices such as input devices 926 including a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc., output devices 927 including a liquid crystal display (LCD), speakers, vibrators, etc., storage devices 928 including magnetic tape, hard disks, etc., and communication devices 929 can be connected to the I / O interface 925. The communication devices 929 can enable the electronic device 920 to communicate wirelessly or via wires with other electronic devices to exchange data. While FIG. 6 shows the electronic device 920 having a variety of devices, it is not required that the electronic device 920 implement or include all of the devices shown, and the electronic device 920 can alternatively implement or include more or fewer devices.

[0070] For example, according to embodiments of the present disclosure, the image generation method described above can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program including program code for executing the image generation method described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 929, or installed from a storage device 928, or installed from a ROM 922. When the computer program is executed by a processing device 921, it can realize functions limited to the image generation method provided by embodiments of the present disclosure.

[0071] 7 is a schematic diagram of a storage medium according to some embodiments of the present disclosure. For example, as shown in FIG. 7, storage medium 930 may be a non-transitory computer-readable storage medium for storing non-transitory computer-executable instructions 931. When the non-transitory computer-executable instructions 931 are executed by a processor, the image generation method described in the embodiments of the present disclosure can be realized, for example, when the non-transitory computer-executable instructions 931 are executed by a processor, one or more steps of the image generation method described above can be performed.

[0072] For example, the storage medium 930 may be applied to the electronic device described above, and for example, the storage medium 930 may include a memory in the electronic device.

[0073] For example, the storage medium may include a memory card in a smartphone, a storage component in a tablet PC, a hard disk in a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or may be other suitable storage media.

[0074] For example, the description of the storage medium 930 may refer to the description of the memory in the embodiment of the electronic device, and will not be repeated here. The specific functions and technical effects of the storage medium 930 may refer to the description of the image generation method described above, and will not be described in further detail here.

[0075] It should be noted that, in the context of the present disclosure, a computer-readable medium may be a tangible medium that can contain or store a program used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable medium may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the above. The computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program, which may be used by or in combination with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a propagated data signal, in baseband or as part of a carrier wave, having computer-readable program code carried therein. Such propagated data signals may take various forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which is capable of transmitting, propagating, or transporting a program for use by or in connection with an instruction execution system, apparatus, or device.The program code contained in the computer readable medium may be transmitted over any suitable medium, including, but not limited to, electrical wire, optical cable, RF (radio frequency), etc., or any suitable combination of the above.

[0076] Other embodiments of the present disclosure will be readily suggested to those skilled in the art in light of the examples of the present disclosure. The present disclosure is intended to cover any modifications, uses, or adaptations of the present disclosure, which modifications, uses, or adaptations follow the general principles of the present disclosure and include common knowledge or conventional technical means known in the art that are not disclosed in the present disclosure. The present disclosure and examples should be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0077] It is to be understood that the present disclosure is not limited to the exact construction already described above and illustrated in the drawings, and that various modifications and changes can be made without departing from the scope thereof, which is limited only by the appended claims.

Claims

1. 1. A method for generating an image, comprising: obtaining a target text, a sketch image and a noise image; obtaining a fusion vector based on the target text and the sketch image; and performing a denoising process on the noise image based on the fusion vector to obtain a target image that matches the target text and the sketch image.

2. obtaining a fusion vector based on the target text and the sketch image, obtaining a first feature vector corresponding to the target text and a second feature vector corresponding to the sketch image; The image generating method according to claim 1 , further comprising: performing information fusion on the first feature vector and the second feature vector to obtain the fusion vector.

3. The information fusion of the first feature vector and the second feature vector includes: The image generation method of claim 2 , further comprising: information fusing the first feature vector and the second feature vector using an attention mechanism.

4. performing a denoising process on the noise image based on the fusion vector to obtain a target image matching the target text and the sketch image; The image generating method according to any one of claims 1 to 3, further comprising: performing a multi-step noise removal operation using a target model to remove noise from the noise image, wherein the multi-step noise removal operation includes at least one first operation of performing a noise removal process based on the fusion vector.

5. The first operation is determining an image to be denoised; if the first operation is a first operation of the multiple-step denoising operation, the image to be denoised is the noise image; if the first operation is not a first operation, the image to be denoised is a processing result of a previous step denoising operation; determining a first time vector corresponding to the first operation; The image generating method according to claim 4 , further comprising: performing a denoising process on the image to be denoised based on the fusion vector and the first time vector using the target model.

6. The step of performing a denoising process on the image to be denoised based on the fusion vector and the first time vector using the target model includes: obtaining a first condition vector based on at least the fusion vector and the first time vector; The image generating method according to claim 5 , further comprising: performing a denoising process on the image to be denoised based on the first condition vector and the fusion vector using the target model.

7. The method further includes obtaining a first feature vector corresponding to the target text, and performing a transformation process on the first feature vector to obtain a target vector; 5. The image generating method according to claim 4, wherein the multiple-step noise removal operation includes at least one step of a second operation for performing noise removal processing based on the target vector.

8. The second operation is determining a second time vector corresponding to the image to be denoised and the second operation; The image generating method according to claim 7 , further comprising: performing a denoising process on the image to be denoised based on the target vector and the second time vector using the target model.

9. 8. The image generating method of claim 7, wherein the performing the multiple-step noise removal operation includes first performing the first operation a predetermined number of steps, and then performing the second operation after the first operation has been completed.

10. 1. An image generation device, comprising: an acquisition module configured to acquire the target text, the sketch image, and the noise image; a fusion module configured to obtain a fusion vector based on the target text and the sketch image; a denoising module configured to perform a denoising process on the noisy image based on the fusion vector to obtain a target image that matches the target text and the sketch image.

11. A computer-readable storage medium having a computer program stored thereon, the computer program causing the computer to execute the image generating method according to any one of claims 1 to 9 when the computer program is executed by the computer.

12. 10. An electronic device comprising a memory and a processor, wherein executable code is stored in the memory, and wherein, when the processor executes the executable code, the electronic device realizes the image generation method according to any one of claims 1 to 9.

13. A computer program which, when executed by a processor, implements the image generation method according to any one of claims 1 to 9.

Images