Image generation method, apparatus, and electronic device
The image generation method integrates text and sketch inputs to enhance the accuracy of image creation, addressing the limitations of text-based systems by using models for feature extraction and denoising to produce images that meet user expectations.
Patent Information
- Application Number
- JP2025536156
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2022-12-19
- Filing Date
- 2023-12-08
- Publication Date
- 2026-11-19
- Estimated Expiration
- 2043-12-08
AI Technical Summary
Current text-based image generation technologies face limitations in accurately describing complex objects such as shape, position, or posture, making it difficult to meet user needs, especially when linguistic descriptions are inadequate.
An image generation method that combines user-inputted text information and simple outline sketches to guide image generation, using models for text feature extraction, information fusion, and denoising operations to create images that match user intent.
Enhances the image generation process by ensuring the produced images align with user needs, overcoming the limitations of linguistic descriptions and improving the accuracy of complex object representation.
Smart Images

Figure 0007954302000001 
Figure 0007954302000002 
Figure 0007954302000003
Abstract
Description
TECHNICAL FIELD
[0001] This application claims the priority of Chinese Patent Application No. 202211634290.6 filed on December 19, 2022, and the entire content disclosed in the above Chinese patent application is incorporated herein by reference as part of this application.
[0002] The present disclosure relates to an image generation method, apparatus, and electronic device.
BACKGROUND ART
[0003] With the application and development of artificial intelligence technology in the image field, image generation technology has emerged as the times require. Currently, there is a demand for generating images purposefully according to user needs. With the continuous development of multimodal technology, text-based image generation technology is advancing rapidly and has achieved excellent effects. However, there are limitations in describing images based on linguistic text, making it difficult to meet user needs.
SUMMARY OF THE INVENTION
[0004] The present disclosure provides an image generation method, apparatus, and electronic device.
[0005] In a first aspect, the present disclosure provides an image generation method, acquiring a target text, a sketch image, and a noise image; obtaining a fusion vector based on the target text and the sketch image; performing denoising processing on the noise image based on the fusion vector to obtain a target image matching the target text and the sketch image.
[0006] In a second aspect, the present disclosure provides an image generation apparatus, an acquisition module configured to acquire a target text, a sketch image, and a noise image; a fusion module configured to obtain a fusion vector based on the target text and the sketch image; The system includes a denoising module for performing denoising on the noise image based on the fusion vector to obtain a target image that matches the target text and the sketch image.
[0007] In a third embodiment, the Disclosure provides a computer-readable storage medium in which a computer program is stored and which is executed by a processor, thereby realizing the method described in any one of the first embodiments.
[0008] In a fourth embodiment, the Disclosure provides an electronic device comprising memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the device implements the method described in any one of the first embodiments.
[0009] Please understand that the general descriptions above and the detailed descriptions below are for illustrative and illustrative purposes only and do not limit this disclosure. [Brief explanation of the drawing]
[0010] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the following briefly introduces the drawings that may be used in describing the embodiments. Clearly, the drawings in the following description are merely some of the embodiments described in this disclosure, and those skilled in the art can obtain further drawings based on these without any creative work.
[0011] [Figure 1] This is a schematic diagram of an image generation scene according to one exemplary embodiment of the present disclosure. [Figure 2] This is a flowchart of an image generation method according to one exemplary embodiment of the present disclosure. [Figure 3] This is a schematic diagram of an example of image generation according to one exemplary embodiment of the present disclosure. [Figure 4] This is a block diagram of an image generation device according to one exemplary embodiment of the present disclosure. [Figure 5] This is a schematic block diagram of an electronic device according to some embodiments of the present disclosure. [Figure 6] This is a schematic block diagram of other electronic devices according to some embodiments of the present disclosure. [Figure 7] This is a schematic diagram of a storage medium according to some embodiments of the present disclosure. [Modes for carrying out the invention]
[0012] To enable those skilled in the art to better understand the technical solutions in this disclosure, the technical solutions in the embodiments of this disclosure will be described clearly and completely below with reference to the drawings of this disclosure. Clearly, the embodiments described are only a selection of embodiments of this disclosure, not all embodiments. All other embodiments obtained based on the embodiments of this disclosure without creative effort by those skilled in the art should also fall within the scope of this disclosure.
[0013] Where the following description relates to drawings, unless otherwise stated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure detailed in the appended claims.
[0014] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit this disclosure. The singular forms “one,” “the said,” and “the said” used in this disclosure are also intended to include many unless the context clearly indicates otherwise. The term “and / or” as used herein should be understood to refer to any or all possible combinations of one or more related enumerated items, and to include these combinations.
[0015] The terms First, Second, Third, etc., may be used in this disclosure to describe various types of information, but it should be understood that such information should not be limited to these terms. These terms are used solely to distinguish information of the same kind from one another. For example, without departing from the scope of this disclosure, First Information may be referred to as Second Information, and similarly, Second Information may be referred to as First Information. Depending on the context, the term “if” as used herein may be interpreted as “when,” “in the event of,” or “in response to a determination.”
[0016] With the application and development of artificial intelligence technology in the field of image processing, image generation technology is emerging at just the right time. Currently, there is a huge demand for generating images with a specific purpose according to user needs. In related technologies, with the continuous development of multimodal technology, text-based image generation technology is improving rapidly and achieving excellent results. However, there are limitations to describing images based on linguistic text, making it difficult to meet user needs. For example, the image to be generated may include complex target objects (e.g., people or objects) such as shape, position, or posture, which are difficult to describe in text. Specifically, for example, if you want to generate an image of a girl dancing, there are many different dance postures, and especially for people who are not professional dancers, it is difficult to describe the desired dance posture in language, making it difficult to generate the ideal image.
[0017] This disclosure provides an image generation method that guides image generation using user-inputted text information or simple outline sketches as conditions for image generation, thereby enabling the creation of images that meet the user's needs. The objective is to improve the image generation effect, ensure that the generated images match the user's needs, and thereby enable image generation according to the user's will.
[0018] As shown in Figure 1, this is a schematic diagram of an image generation scene according to an exemplary embodiment. Hereinafter, aspects of this disclosure will be schematically described with reference to Figure 1 in relation to one complete and specific application example. This application example will describe a specific image generation process.
[0019] As shown in Figure 1, first, a user can provide text T and a sketch image P as conditions for guiding image generation. Here, the text T may be simple descriptive text for the target image to be generated by the user, and the sketch image P may be the outline and position of the target object in the target image to be generated, which is simply drawn by the user as needed. For example, when the user wants to generate an image of a dancing girl, the text T may include "dancing girl", and the sketch image P may be the outline and dancing posture of the girl simply drawn by the user.
[0020] Then, the text T is input into the model M1 to perform text feature extraction, so as to obtain a text vector Rt corresponding to the text T. Here, the text vector Rt may be, for example, a text embedding vector, and the model M1 may be a model capable of extracting a text embedding vector from text. This embodiment does not limit the specific type of the model M1. At the same time, the sketch image P is input into the encoder E to perform image feature extraction, so as to obtain an image vector Rp corresponding to the sketch image P. Here, the image vector Rp may be an image feature vector that can be information-fused with the text vector, and the encoder E may be an encoder capable of extracting the above-mentioned image feature vector from an image. This embodiment does not limit the specific type of the encoder E.
[0021] Next, by inputting the text vector Rt and the image vector Rp into the model M2, information fusion is performed on the text vector Rt and the image vector Rp, so that a fused vector Rr in which the information of the text T and the information of the sketch image P are fused can be obtained. For ease of understanding, the model M2 may be any model capable of performing information fusion on a vector obtained based on text and a vector obtained based on an image, and the present embodiment does not limit the specific type of the model M2. Optionally, the model M2 may be a deep learning model using an attention mechanism, which enables better information fusion of the text vector Rt and the image vector Rp.
[0022] Further, a noise image S, which may be a randomly generated white Gaussian noise image, is acquired. The fused vector Rr is input into the model M3 together with the noise image S, the model M3 may be a pre-trained diffusion model, and the model M3 can perform a one-step denoising operation on the noise image S based on the fused vector Rr to obtain an intermediate image S1. Since the intermediate image S1 is an image obtained by removing a small portion of noise from the noise image S, the intermediate image S1 still contains a large amount of noise. The intermediate image S1 and the fused vector Rr are re-input into the model M3, and the model M3 can perform a one-step denoising operation on the intermediate image S1 based on the fused vector Rr to obtain an intermediate image S2. Subsequent similar processes are inferred in this manner, and will not be repeated here.
[0023] Optionally, the text vector Rt may be input alone into the model M2, and processed by the model M2 to obtain a vector Rc containing only the information of the text T. After performing n steps of denoising operations to obtain the intermediate image Sn, the intermediate image Sn can be input into the model M3 together with the vector Rc, and the model M3 can perform a one-step denoising operation on the intermediate image Sn based on the vector Rc to obtain an intermediate image Sn+1. Thereafter, through a plurality of steps of similar denoising operations, a target image is obtained. The target image is a clear image that meets the user's needs.
[0024] Furthermore, the encoder E, model M1, model M2, and model M3 can be trained together. Specifically, a sample image may be acquired first, and then a sketch image to be mapped to the sample image may be acquired. For example, the sketch image to be mapped to the sample image may be drawn manually. Alternatively, a sketch image to be mapped to the sample image may be extracted using a predetermined image processing algorithm. Alternatively, the sketch image to be mapped to the sample image may be extracted using a machine learning model. Note that the sketch image to be mapped to the sample image may be extracted using any other suitable method, and this embodiment is not limited in this respect. The text content to be mapped to the sample image may be acquired, and for example, the text content to be mapped to the sample image may be edited manually, or the text content to be mapped to the sample image may be extracted using a machine learning model.
[0025] Next, a multi-step noise addition operation is performed on the sample image to obtain a sample noise image, where each step of the noise addition operation adds a small amount of white Gaussian noise to the sample image. Then, the sample noise image, the text content and sketch image mapped to the sample image are input to the corresponding model to be trained according to the information flow shown in Figure 1, and the prediction loss is calculated based on the output result. The model parameters of the model to be trained are then adjusted based on the prediction loss, thereby completing the model training.
[0026] The present disclosure will be described in detail below with reference to specific examples.
[0027] Figure 2 is a flowchart illustrating an image generation method according to an exemplary embodiment. The implementing entity of the method can be any device, platform, server, or device cluster having computing and processing capabilities. The method may include the following steps:
[0028] As shown in Figure 2, step 201 acquires the target text, sketch image, and noise image.
[0029] In this embodiment, the noise image may be randomly generated using white Gaussian noise, and this embodiment does not limit the specific method for generating the noise image. The target text may be a simple descriptive text for the target image to be generated by the user, and the sketch image may be the outline and position of the target object in the target image to be generated, drawn simply by the user as needed. For example, if the user wants to generate an image of a cat, the target text may include "sleeping lazy cat," and the sketch image may be the outline and sleeping posture of the cat drawn simply by the user. Here, the target text may be obtained through editing by the user, or it may be text content selected by the user from multiple candidate text options. The sketch image may be drawn manually by the user, or it may be a sketch image selected by the user from multiple candidate sketch image options. The target text and sketch image can be obtained by any suitable method, and this embodiment does not limit the specific method for obtaining the target text and sketch image.
[0030] In step 202, a fusion vector is obtained based on the target text and sketch image.
[0031] In this embodiment, a fusion vector can be obtained based on target text and a sketch image. In one embodiment, first, the target text is converted into a reference image using a machine learning model, then image features of the reference image and the sketch image are extracted, and the image features of the reference image and the sketch image are fused to obtain a fusion vector.
[0032] In other embodiments, a text vector corresponding to the target text is obtained as the first feature vector, and an image vector corresponding to the sketch image is obtained as the second feature vector. The first and second feature vectors are then merged to obtain a fused vector. Selectively, the first and second feature vectors can be merged using an attention mechanism. Specifically, first, a language model pre-trained with image-text matching data converts the target text into a text vector as the first feature vector, and then a pre-trained encoder processes the sketch image to obtain an image vector as the second feature vector. The first and second feature vectors are input to a deep learning model based on an attention mechanism, and the deep learning model merges the first and second feature vectors to obtain a fused vector. This embodiment improves the fusion effect by using an attention mechanism to merge the target text and the sketch image.
[0033] In step 203, denoising is performed on the noisy image based on the fusion vector to obtain a target image that matches the target text and sketch image.
[0034] In this embodiment, based on the fusion vector described above, a target image can be obtained by denoising the noise image according to the content described in the target text and the contour drawn in the sketch image. As shown in Figure 3, for example, the text content of the target text 301 may include "red rose," and the sketch image 302 may be the contour of a rose simply drawn by the user. Based on the fusion vector of the target text and the sketch image, a target image 303 can be obtained by denoising the noise image. The target image 303 matches the description in the target text 301 and the contour in the sketch image 302.
[0035] In one embodiment, a target image can be obtained by performing a one-step denoising process on a noisy image based on a fusion vector using a machine learning model or a preset algorithm.
[0036] In other embodiments, noise reduction can be performed on a noisy image by a multi-step denoising operation using a pre-trained target model, which may be a diffusion model. Here, the multi-step denoising operation may include at least one first operation, which is an operation that guides the denoising process based on the target text and sketch image (i.e., an operation that performs denoising based on a fusion vector).
[0037] Selectively, the multi-step denoising operation may include at least one second step, the second step being an operation that guides the denoising process based only on the target text (i.e., an operation that performs denoising based on a target vector corresponding to the target text). Specifically, as the inventors discovered in several experiments, performing denoising based on both the target text and the sketch image is more prone to overfitting, meaning the sketch image imposes too much constraint on the target image. Therefore, the inventors conceived of the multi-step denoising operation, which can also be based only on the target text, in addition to using both the target text and the sketch image as conditions, thereby solving the overfitting problem in the denoising process.
[0038] To make it easier to understand, in a multi-step denoising operation, the first and second operations may be performed alternately. Optionally, the second operation may be performed after the first operation is completed. By performing the first operation first, which is conditional on the target text and sketch image, the sketch image can be better constrained in forming the contours of the target object in the target image, and the target text can be more constrained in forming the details in the target image. This not only avoids the problem of overfitting but also further improves the quality of the resulting image.
[0039] Specifically, the first operation of any step may include the following operations: First, the image to be denoised is determined, and here, if the first operation is the first operation of a multi-step denoising operation, the acquired noise image may be used as the image to be denoised. If the first operation is not the first operation, the processing result of the denoising operation in the previous step may be used as the image to be denoised. To make it clear, the denoising operation in the previous step may be the first operation or the second operation.
[0040] Next, a first time vector corresponding to the first operation is determined, and denoising is performed on the image to be denoised based on the fused vector and the first time vector using the target model. Here, the first time vector (e.g., time embedding) may be a vector that can characterize the time in the multi-step denoising operation of the first operation. For example, the first operation is the nth step in the multi-step denoising operation, and in this case, the first time vector corresponding to the first operation is a vector that can characterize the nth step. Specifically, the fused vector and the first time vector can be merged (e.g., added) to obtain a first condition vector. Next, denoising is performed on the image to be denoised based on the first condition vector and the fused vector using the target model. Note that in addition to the fused vector, other feature vectors can also be merged with the time vector along with the fused vector to obtain a condition vector, and this embodiment is not limited to this.
[0041] For the second operation, before performing the second operation, a text vector corresponding to the target text is obtained as the first feature vector, and the target vector is obtained by performing a transformation process on the first feature vector. The second operation of any step may include the following operations: First, the image to be denoised and the second time vector corresponding to the second operation are determined. Here, if the second operation is the first operation of a multi-step denoising operation, the obtained noise image can be used as the image to be denoised. If the second operation is not the first operation, the processing result of the denoising operation in the previous step can be used as the image to be denoised. To make it clear, the denoising operation in the previous step may be the first operation or the second operation.
[0042] Next, using the target model, denoising is performed on the image to be denoised based on the target vector and the second time vector. Specifically, the target vector and the second time vector can be merged to obtain the second condition vector. Subsequently, using the target model, denoising is performed on the image to be denoised based on the second condition vector and the target vector.
[0043] In this embodiment, a specific application scenario might be that a user wants to create a piece of artwork. They could easily sketch the artwork as they envision it, convert the sketch into a preset image format, and edit a simple text description. Here, the sketch would include the shape, position, and contours of the target object. Based on the sketch and text description, the system can guide the generation of a target image that reflects the user's ideas.
[0044] Another specific application scenario is that during the process of solving a crime, a witness may select the sketch that most closely resembles the suspect's face and physique from several candidates, or the witness may draw a sketch of the suspect's face and physique themselves and then edit a text description detailing the suspect's features. Using the sketch or text description as a guide, an image that closely resembles the suspect can be generated.
[0045] This embodiment can be applied not only to the application scenes described above but also to other scenes. The image generation method provided in this disclosure obtains a fusion vector based on target text and a sketch image, performs noise reduction processing on a noise image obtained based on the fusion vector, and obtains a target image to be mapped to the target text and sketch image. In this embodiment, a sketch image having spatial information (difficult to describe in language) and target text including detailed feature descriptions (difficult to draw) are fused, thereby increasing the connectivity between spatial information and text information, making the noise-reduced target image match the user's needs and better satisfying those needs. This improves the image generation effect and achieves the objective of generating images according to the user's will.
[0046] Corresponding to the embodiments of the image generation method described above, this disclosure further provides embodiments of an image generation apparatus.
[0047] As shown in Figure 4, Figure 4 is a block diagram of an image generation apparatus according to an exemplary embodiment of the present disclosure, which may include an acquisition module 401, a fusion module 402, and a noise reduction module 403.
[0048] Here, acquisition module 401 is used to acquire the target text, sketch image, and noise image.
[0049] The fusion module 402 is used to obtain a fusion vector based on the target text and the sketch image described above.
[0050] The noise reduction module 403 is used to perform noise reduction processing on the noise image based on the fusion vector in order to obtain a target image that matches the target text and sketch image.
[0051] In some embodiments, the fusion module 402 may include an acquisition submodule and a fusion submodule (not shown).
[0052] Here, the acquisition submodule is used to acquire a first feature vector corresponding to the target text and a second feature vector corresponding to the sketch image.
[0053] The fusion submodule is used to obtain a fused vector by fusing the first feature vector and the second feature vector.
[0054] In some other embodiments, the fusion submodule is configured to information-fuse the first feature vector and the second feature vector using an attention mechanism.
[0055] In some other embodiments, the noise reduction module 403 may include an execution submodule (not shown).
[0056] Here, the execution submodule is used to denoise a noisy image by performing a multi-step denoising operation using a pre-trained target model. Here, the multi-step denoising operation includes at least one first step that performs denoising based on a fusion vector.
[0057] In some other embodiments, the execution submodule may perform any step of the first operation to determine the image to be denoised. If the first operation is the first operation of the multi-step denoising operation, the image to be denoised is the noisy image. If the first operation is not the first operation, the image to be denoised is the result of the denoising operation of the previous step. A first time vector corresponding to the first operation is determined, and denoising is performed on the image to be denoised based on the fusion vector and the first time vector using the target model.
[0058] In other embodiments, the execution submodule can perform the following: Using the target model, denoise the image to be denoised based on the fusion vector and the first time vector to obtain a first condition vector based on at least the fusion vector and the first time vector. Using the target model, denoise the image to be denoised based on the first condition vector and the fusion vector.
[0059] In some other embodiments, the acquisition module 401 is further used to acquire a first feature vector corresponding to the target text and to transform the first feature vector to obtain a target vector. Here, the multi-step denoising operation includes at least one second step of denoising based on the target vector.
[0060] In some other embodiments, the execution submodule can perform any step of the second operation as follows: determine the image to be denoised and the second time vector corresponding to the second operation; and perform denoising on the image to be denoised based on the target vector and the second time vector using the target model.
[0061] In some other embodiments, the execution submodule may first perform a first operation in predetermined steps, and then perform a second operation after the completion of the first operation.
[0062] For embodiments of the apparatus, refer to the partial description of the embodiment of the method for the relevant parts, as these embodiments substantially correspond to the embodiments of the method. The embodiments of the apparatus described above are merely schematic, and the units shown here as separate parts may or may not be physically separated, and the parts shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Depending on the actual needs, some or all of these modules can be selected to achieve the objectives of the technical proposal of the embodiments of this disclosure. Those skilled in the art will be able to understand and implement these without any creative effort.
[0063] Figure 5 is a schematic block diagram of an electronic device according to some embodiments of the present disclosure. As shown in Figure 5, the electronic device 910 includes a processor 911 and a memory 912, which may be used to implement a client or a server. The memory 912 is used to non-temporarily store computer executable instructions (e.g., one or more computer program modules). The processor 911 is used to execute the computer executable instructions, which, when executed by the processor 911, can perform one or more steps of the image generation method described above to implement the image generation method described above. The memory 912 and the processor 911 can be interconnected by a bus system and / or other forms of connection mechanisms (not shown).
[0064] For example, the processor 911 may be a central processing unit (CPU), a graphics processing unit (GPU), or another type of processing unit having data processing capability and / or program execution capability. For example, the central processing unit (CPU) may be an X86 or ARM architecture, etc. The processor 911 may be a general-purpose processor or a dedicated processor, and can control other components in the electronic device 910 to perform desired functions.
[0065] For example, memory 912 may include any combination of one or more computer program products that can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or high-speed cache memory (cache). Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules can be stored in the computer-readable storage media, and the processor 911 can execute one or more computer program modules to realize various functions of the electronic device 910. The computer-readable storage media may also store various applications and various data, as well as various data used and / or generated by the applications.
[0066] In the embodiments of this disclosure, the specific functions and technical effects of the electronic device 910 are described in the above-described image generation method and will not be described in further detail here.
[0067] Figure 6 is a schematic block diagram of another electronic device according to some embodiments of the present disclosure. The electronic device 920 is suitable, for example, for carrying out an image generation method provided by an embodiment of the present disclosure. The electronic device 920 may be a terminal device or the like, and may be used to realize a client or server. The electronic device 920 includes, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet PCs), PMPs (portable multimedia players), in-vehicle terminals (e.g., car navigation terminals), and wearable electronic devices, and fixed terminals such as digital TVs, desktop computers, and smart home devices. Note that the electronic device 920 shown in Figure 6 is merely an example and does not impose any limitations on the functionality and scope of use of the embodiments of the present disclosure.
[0068] As shown in Figure 6, the electronic device 920 may include a processing unit (e.g., a central processing unit, an image processing unit, etc.) 921 which can perform various appropriate operations and processes according to programs stored in read-only memory (ROM) 922 or programs loaded from storage device 928 into random access memory (RAM) 923. RAM 923 also stores various programs and data necessary for the operation of the electronic device 920. The processing unit 921, ROM 922, and RAM 923 are connected to each other via bus 924. An input / output (I / O) interface 925 is also connected to bus 924.
[0069] Typically, devices such as input devices 926 including touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, and gyroscopes; output devices 927 including liquid crystal displays (LCDs), speakers, and vibrators; storage devices 928 including magnetic tape and hard disks; and communication devices 929 can be connected to the I / O interface 925. The communication device 929 can enable the electronic device 920 to exchange data with other electronic devices via wireless or wired communication. Figure 6 shows an electronic device 920 with various devices, but it is not necessary to implement or have all of the devices shown, and the electronic device 920 may alternatively implement or have more or fewer devices.
[0070] For example, according to embodiments of the present disclosure, the image generation method described above can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product which includes a computer program carried on a non-temporary computer-readable medium, the computer program which includes program code for executing the image generation method described above. In such embodiments, the computer program can be downloaded and installed from a network by a communication device 929, or installed from a storage device 928, or installed from a ROM 922. When the computer program is executed by a processing device 921, it can realize functions limited to the image generation method provided by embodiments of the present disclosure.
[0071] Figure 7 is a schematic diagram of a storage medium according to some embodiments of the present disclosure. For example, as shown in Figure 7, the storage medium 930 may be a non-temporary computer-readable storage medium for storing non-temporary computer-executable instructions 931. When the non-temporary computer-executable instructions 931 are executed by a processor, the image generation method described in embodiments of the present disclosure can be realized, for example, when the non-temporary computer-executable instructions 931 are executed by a processor, one or more steps of the image generation method described above can be performed.
[0072] For example, the storage medium 930 may be applied to the electronic device described above, and for example, the storage medium 930 may include memory in the electronic device.
[0073] For example, the storage medium may include a smartphone memory card, a tablet PC memory component, a personal computer hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.
[0074] For example, a description of the storage medium 930 can be found in the description of memory in the embodiment of the electronic device, and will not be repeated here. The specific functions and technical effects of the storage medium 930 can be found in the description of the image generation method described above, and will not be described in further detail here.
[0075] In the context of this disclosure, a computer-readable medium may be a tangible medium that contains or stores a program used by or in combination with an instruction execution system, apparatus, or device. A computer-readable medium may be a computer-readable signal medium, a computer-readable storage medium, or any combination of both. A computer-readable storage medium may, but are not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of more than these. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program, which may be used by or in combination with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signaling medium may include data signals propagated in the baseband or as part of a carrier wave, wherein computer-readable program code is carried therein. Such propagated data signals may take various forms and include, but are not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signaling medium may be any computer-readable medium other than a computer-readable storage medium, and such computer-readable signaling medium may transmit, propagate, or transmit programs used by or in combination with an instruction execution system, apparatus, or device.Program code contained in a computer-readable medium can be transmitted on any suitable medium, which includes, but is not limited to, electric wires, optical cables, RF (radio frequency), or any suitable combination thereof.
[0076] Other embodiments of the Disclosure will be readily conceivable to those skilled in the art, given the examples of the Disclosure. The Disclosure is intended to encompass all variations, uses, or adaptive changes of the Disclosure, including common or customary technical means known in the non-disclosed art, in accordance with the general principles of the Disclosure. The Disclosure and its examples should be considered illustrative only, and the true scope and spirit of the Disclosure are indicated by the claims.
[0077] This disclosure is not limited to the precise structure described above and shown in the drawings, and it should be understood that various modifications and changes may be made as long as they do not deviate from that scope. The scope of this disclosure is limited only by the attached claims.
Claims
1. An image generation method, To acquire target text, sketch images, and noise images, Obtain a first feature vector corresponding to the target text, and perform a transformation process on the first feature vector to obtain a target vector. Based on the target text and the sketch image, a fusion vector is obtained, This includes performing noise reduction processing on the noise image based on the fusion vector to obtain a target image that matches the target text and the sketch image, Performing noise reduction processing on the noise image based on the aforementioned fusion vector to obtain a target image that matches the target text and the sketch image is, An image generation method comprising performing a noise reduction operation on a noisy image using a target model, wherein the multiple steps of the noise reduction operation include at least one first step of performing a noise reduction operation based on the fusion vector, and at least one second step of performing a noise reduction operation based on the target vector.
2. Obtaining a fusion vector based on the aforementioned target text and sketch image is possible. Obtaining a first feature vector corresponding to the target text and a second feature vector corresponding to the sketch image, The image generation method according to claim 1, comprising information fusion of the first feature vector and the second feature vector to obtain the fused vector.
3. The information fusion of the first feature vector and the second feature vector described above is The image generation method according to claim 2, comprising information fusion of the first feature vector and the second feature vector using an attention mechanism.
4. The first operation is, The process involves determining the image to be denoised, and if the first operation is the first operation of the multi-step denoising process, the image to be denoised is the noisy image, and if the first operation is not the first operation, the image to be denoised is the result of the denoising process in the previous step. Determining the first time vector corresponding to the first operation, The image generation method according to claim 1, further comprising using the target model and performing noise reduction processing on the image to be denoised based on the fusion vector and the first time vector.
5. Using the aforementioned target model, performing noise reduction processing on the image to be denoised based on the fusion vector and the first time vector is: At least the first condition vector is obtained based on the fusion vector and the first time vector, The image generation method according to claim 4, further comprising using the target model to perform noise reduction processing on the image to be denoised based on the first condition vector and the fusion vector.
6. The second operation described above is, Determine the image to be denoised and the second time vector corresponding to the second operation, The image generation method according to claim 1, further comprising using the target model and performing noise reduction processing on the image to be denoised based on the target vector and the second time vector.
7. The image generation method according to claim 1, wherein the execution of the multi-step noise reduction operation described above includes first performing the first operation in predetermined steps, and then performing the second operation after the completion of the first operation.
8. An image generation device, An acquisition module configured to acquire target text, a sketch image, and a noise image, and further configured to acquire a first feature vector corresponding to the target text, and to transform the first feature vector to obtain a target vector, A fusion module configured to obtain a fusion vector based on the target text and the sketch image, The denoising module is configured to perform denoising on the noise image based on the fusion vector to obtain a target image that matches the target text and the sketch image, The noise reduction module includes an execution submodule that performs a noise reduction operation on the noisy image using a target model, wherein the multiple steps of the noise reduction operation include at least one first step of performing noise reduction based on the fusion vector, and at least one second step of performing noise reduction based on the target vector, in an image generation apparatus.
9. A computer-readable storage medium in which a computer program is stored, wherein when the computer program is executed on a computer, the computer causes the computer to execute the image generation method described in any one of claims 1 to 7.
10. An electronic device comprising a memory and a processor, wherein executable code is stored in the memory, and when the processor executes the executable code, the image generation method according to any one of claims 1 to 7 is realized.
11. A computer program that, when executed by a processor, realizes the image generation method described in any one of claims 1 to 7.
Citation Information
Patent Citations
Image generation model training method and image generation method
CN110163267A
Method, device and equipment for generating virtual clothing through multi-modal fusion and storage medium
CN114723843A
Image generation method and device and electronic equipment
CN115018954A
Image generation method, and training method and device of image denoising model
CN115222630A
Variational auto-encoder for outputting a 3D model
EP3958182A1
