Image generation method, apparatus, computer-readable storage medium, electronic device, and computer program
The image generation method addresses the limitations of existing technologies by using reference images to extract and process semantic feature vectors, enabling the creation of images with consistent content and style through denoising and super-resolution, thereby enriching image generation capabilities.
Patent Information
- Application Number
- JP2025534943
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2022-12-19
- Filing Date
- 2023-12-12
- Publication Date
- 2026-11-19
- Estimated Expiration
- 2043-12-12
AI Technical Summary
Current image generation technologies, such as GAN and VAE, struggle to generate images according to user needs, particularly in special scenarios where images with similar content, theme, and style are required, limiting their effectiveness.
An image generation method that utilizes a reference image to extract high-level semantic feature vectors, performs denoising and super-resolution processing using diffusion models, and concatenates noise images to produce target images with partially identical semantic information.
Enhances image generation by producing a series of images with consistent high-level semantic information, improving the image generation effect and providing a richer range of image generation methods.
Smart Images

Figure 0007954300000001 
Figure 0007954300000002 
Figure 0007954300000003
Abstract
Description
TECHNICAL FIELD
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the priority of Chinese Patent Application No. 202211633511.8 filed on December 19, 2022, the entire content of the Chinese patent application disclosed herein is incorporated as part of the present application.
[0002] The present disclosure relates to an image generation method, apparatus, computer-readable storage medium, electronic device, and computer program. BACKGROUND ART
[0003] With the application and development of artificial intelligence technology in the image field, image generation technology has emerged. Currently, there is a strong demand for intentionally generating images according to user needs. In related art, images are generally generated based on technologies such as GAN and VAE, but it is difficult to generate images according to user needs, and it is difficult to meet the needs of some special scenarios, which has great limitations. Accordingly, there is a current need for a solution capable of generating images according to user needs. SUMMARY OF THE INVENTION
[0004] The present disclosure provides an image generation method, apparatus, computer-readable storage medium, electronic device, and computer program.
[0005] According to a first aspect, an image generation method is provided, the method comprising: acquiring a reference image and a first noise image; acquiring a reference feature vector corresponding to the reference image; performing denoising processing on the first noise image based on the reference feature vector to obtain a target image that is a variation of the reference image, wherein the target image and the reference image share partially identical high-level semantic information.
[0006] According to the second embodiment, an image generation apparatus is provided, the apparatus is An acquisition module for obtaining a reference image and a first noise image, An extraction module for obtaining a reference feature vector corresponding to the aforementioned reference image, A processing module for performing noise reduction processing on the first noise image based on the reference feature vector to obtain a target image after variation of the reference image, wherein the target image and the reference image have high-level semantic information that partially match.
[0007] According to a third embodiment, a computer-readable storage medium is provided, the storage medium stores a computer program, and when the computer program is executed by a processor, the method of the first embodiment is realized.
[0008] According to the fourth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the method of the first aspect is realized when the processor executes the program.
[0009] According to the fifth aspect, a computer program is provided, and when the computer program is executed by a processor, the method of the first aspect is realized.
[0010] Please understand that the above general description and the following detailed description are illustrative and explanatory only and do not limit this disclosure. [Brief explanation of the drawing]
[0011] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the following drawings, which may be used in the description of the embodiments, are briefly introduced below. As will be apparent, the drawings in the following description are only a few of the embodiments described in this disclosure, and further drawings can be obtained based on these drawings, assuming that those skilled in the art do not require any creative work.
[0012] [Figure 1] This is a schematic diagram showing an image generation scenario according to one exemplary embodiment of the present disclosure. [Figure 2] This is a flowchart of an image generation method according to one exemplary embodiment of the present disclosure. [Figure 3] This is a schematic diagram showing an example of image generation according to one exemplary embodiment of the present disclosure. [Figure 4] This is a block diagram of an image generation device according to one exemplary embodiment of the present disclosure. [Figure 5] This is a schematic block diagram of an electronic device according to some embodiments of the present disclosure. [Figure 6] This is a schematic block diagram of other electronic devices relating to some embodiments of the present disclosure. [Figure 7] This is a schematic diagram of a storage medium according to some embodiments of the present disclosure. [Modes for carrying out the invention]
[0013] To enable those skilled in the art to better understand the technical solutions of the disclosure, the technical solutions of the embodiments of the disclosure will be described clearly and completely below with reference to the drawings of the embodiments of the disclosure. It is clear that the embodiments described are only a selection of the embodiments of the disclosure, not all of them. Any other embodiments that a person skilled in the art can obtain based on the embodiments of the disclosure without requiring any creative work will also fall within the scope of the disclosure.
[0014] Where the following description refers to drawings, unless otherwise indicated, the same numbers in different drawings refer to the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure detailed in the appended claims.
[0015] The terms used in this disclosure are for illustrative purposes only and are not intended to limit the disclosure. The singular forms “one kind,” “the said,” and “the said” as used in this disclosure include the plural form unless the context clearly indicates otherwise. Furthermore, the terms “and / or” as used herein should be understood to include any or any possible combination of the one or more items listed relating to the disclosure.
[0016] To ensure understanding, this disclosure may use terms such as 1st, 2nd, 3rd to describe different types of information, but this information should not be limited to these terms. These terms are used solely to distinguish information of the same kind from one another. For example, without departing from the scope of this disclosure, 1st information may be referred to as 2nd information, and similarly, 2nd information may be referred to as 1st information. Depending on the context, the term “if” as used herein may also be interpreted as “in the case of…” or “when…” or “corresponding to a decision of…”
[0017] With the application and development of artificial intelligence technology in the field of image processing, image generation technologies have emerged. Currently, there is a great demand for intentionally generating images according to user needs. Related technologies generally generate images based on techniques such as GAN and VAE, but generating images according to user needs is difficult, and it is difficult to meet the needs of some special scenarios, so there are significant limitations. For example, a designer who has created an image work in their own style may want to obtain a series of images that are similar in content, theme, and style, but it is not possible to generate an ideal series of images based on GAN, VAE, etc.
[0018] This disclosure provides an image generation method that uses a reference image input by the user as an image generation condition, guides the image generation process, and allows obtaining a target image that is partially identical to the reference image and possesses a high level of semantic information. This improves the image generation effect and provides users with a richer range of image generation methods.
[0019] Referring to Figure 1, Figure 1 is a schematic diagram illustrating an image generation scenario according to an exemplary embodiment. Hereinafter, the solution of the present disclosure will be exemplarily described with reference to Figure 1.
[0020] As shown in Figure 1, first, a user can provide a reference image P as a condition for guiding image generation. The reference image P is input into model M11, model M12 ... model M1n respectively, to extract a plurality of different high-level semantic feature vectors corresponding to the reference image P, and the plurality of high-level semantic feature vectors may include vector Rp1, vector Rp2 ... vector Rpn. A concatenation operation is performed on the plurality of high-level semantic feature vectors to obtain a reference vector Rp. Here, model M11, model M12 ... model M1n are models of the same type but with different parameters, and these models can extract corresponding feature vectors for different high-level semantics in the image.
[0021] Further, a noise image S is acquired, and the noise image S may be a randomly generated Gaussian white noise image. The reference vector Rp is input into model M2 together with the noise image S. Model M2 may be a pre-trained diffusion model, and model M2 performs a one-step denoising operation on the noise image S based on the reference vector Rp to obtain an image S1. The image S1 is an image obtained by removing a small part of noise from the noise image S, and therefore the image S1 still contains a large amount of noise. The image S1 and the reference vector Rp are input into model M2, and model M2 performs a one-step denoising operation on the image S1 based on the reference vector Rp to obtain an image S2. Subsequent similar processes can be deduced in the same manner, and thus will not be described again here. Finally, after denoising processing by a plurality of model M2, an initial image A can be obtained. The initial image A may be an image with a relatively low resolution, for example, may be an image with a resolution of 64*64.
[0022] Subsequently, a noise image B is acquired, and the noise image B may also be a randomly generated Gaussian white noise image. First, the initial image A is upsampled to obtain an image with a higher resolution than the initial image A. Next, the image obtained by upsampling and the noise image B are subjected to a concatenation process to obtain an image A1 to be processed. The image A1 to be processed is input to the model M31. The model M31 may be a diffusion model for performing super-resolution processing on images. The model M31 performs a one-step denoising operation on the image A1 to obtain an image A2. Next, the image A2 is input to the model M31, and the model M31 performs a one-step denoising operation on the image A2 to obtain an image A3. Since the subsequent similar processes can be inferred in the same manner, they will not be described again here. Finally, after denoising processing by a plurality of models M31, an intermediate image C can be obtained. The resolution of the intermediate image C is higher than the resolution of the initial image A. For example, the intermediate image C may be an image with a resolution of 256*256.
[0023] Finally, a noise image D of randomly generated Gaussian white noise is acquired. First, the intermediate image C is upsampled to obtain an image with a higher resolution than the intermediate image C. Next, the image obtained by upsampling and the noise image D are subjected to a concatenation process to obtain an image C1 to be processed. The image C1 to be processed is input to the model M32. The model M32 may also be a diffusion model for performing super-resolution processing on images. The model M32 performs a one-step denoising operation on the image C1 to obtain an image C2. Next, the image C2 is input to the model M32, and the model M32 performs a one-step denoising operation on the image C2 to obtain an image C3. Since the subsequent similar processes can be inferred in the same manner, they will not be described again here. Finally, after denoising processing by a plurality of models M32, a target image can be obtained. The resolution of the target image is higher than the resolution of the intermediate image C. For example, the target image may be an image with a resolution of 1024*1024.
[0024] Furthermore, models M11, M12…M1n, M2, M31, and M32 may be trained individually, simultaneously, or roughly individually before being trained together. Specifically, first, a sample image is acquired, and then a matching image corresponding to the sample image is acquired. For example, the matching image corresponding to the sample image may be drawn manually or using Adobe Photoshop® software. As another example, the matching image corresponding to the sample image may be drawn using a pre-configured image processing algorithm. As yet another example, a series of image materials may be searched, and the sample image and matching image may be selected from them. It should be understood that the matching image corresponding to the sample image may be acquired by any other reasonable method, and this embodiment is not limited to this point.
[0025] Next, following the information flow shown in Figure 1, a multi-step noise addition operation is performed on the sample image in the reverse direction, followed by downsampling, and then another multi-step noise addition operation is performed after downsampling. A small amount of Gaussian white noise is added through the noise addition operation at each step, and finally a sample noise image is obtained. Next, the sample noise image and the sample image are input into the appropriate model to be trained, respectively, following the information flow shown in Figure 1, the prediction loss is calculated based on the output result, and the model parameters of the model to be trained are adjusted based on the prediction loss, thereby completing the training of the model.
[0026] The present disclosure will be described in detail below with reference to specific examples.
[0027] Figure 2 is a flowchart of an image generation method according to one exemplary embodiment. The execution body of this method can be implemented as any device, platform, server, or device cluster having computing and processing capabilities. The method includes the following steps.
[0028] As shown in Figure 2, in step 201, the reference image and the first noise image are acquired.
[0029] In this embodiment, the first noise image may be an image randomly generated with Gaussian white noise, and this embodiment is not limited to a specific method for generating the first noise image. The reference image may be a comparison image provided by the user, and the target image can be generated by referencing some high-level semantic information from the reference image. For example, the user may create an image of a cat playing with a ball in advance, use this image as the reference image, and generate a series of variant images related to the cat playing with the ball, in which the cat's posture may differ in the variant images.
[0030] In step 202, we obtain the reference feature vector corresponding to the reference image.
[0031] In one implementation, a single pre-trained first model may be used to extract high-level semantic feature vectors corresponding to a reference image as the reference feature vector. In another implementation, multiple different first models may be used to extract multiple different high-level semantic feature vectors corresponding to the reference image, and these multiple different high-level semantic feature vectors may be concatenated to obtain the reference feature vector. Here, multiple different first models can extract feature vectors appropriate to different high-level semantics in the image. In this implementation, multiple different first models are used to extract multiple different high-level semantic feature vectors corresponding to the reference image, and these multiple different high-level semantic feature vectors are concatenated. This improves the effectiveness of extracting the style content of the reference image and increases the correlation between the generated target image and the high-level semantics of the reference image.
[0032] In step 203, denoising is performed on the first noise image based on the reference feature vector to obtain a target image after variation of the reference image, and the target image and the reference image have high-level semantic information in which parts are the same.
[0033] In this embodiment, based on the above reference feature vector, noise reduction processing is performed on the noise image according to the high-level semantic features of the reference image to obtain the target image. As shown in Figure 3, the reference image 301 may be a source image provided by the user, and the image content is specifically a bridge over a river. By performing noise reduction processing on different noise images based on the reference image 301, different target images can be obtained. For example, by performing noise reduction processing on the noise image 302 based on the reference image 301, target image 303 can be obtained. By performing noise reduction processing on the noise image 304 based on the reference image 301, target image 305 can be obtained. Here, both the noise image 302 and the noise image 304 are randomly generated Gaussian white noise.
[0034] Specifically, in one implementation, a machine learning model or a pre-configured algorithm can be used to perform a one-step denoising process on the first noisy image based on a reference feature vector to obtain the target image. In another implementation, a pre-trained second model can be used based on the reference feature vector to perform a multi-step denoising operation on the first noisy image to obtain the target image, and this second model may be a diffusion model.
[0035] In another implementation, a first noisy image is denoised by performing a multi-step denoising operation using a second model based on a reference feature vector, thereby obtaining an initial image with relatively low resolution. Next, a super-resolution process is performed on the initial image using at least one third model to obtain a target image with relatively high resolution, and this third model may also be a diffusion model.
[0036] Specifically, on the one hand, using the second model to perform a denoising operation at any step in a multi-step operation may include the following operations: First, the image to be denoised is determined, and if this is the first denoising operation, the image to be denoised is the first noisy image. If this is not the first denoising operation, the image to be denoised is the result of the denoising operation in the previous step.
[0037] Next, a timing vector corresponding to the denoising operation is determined, and denoising is performed on the image to be denoised using the second model, based on the reference feature vector and the timing vector. The timing vector may be a vector that can characterize the timing of the denoising operation in the multi-step denoising operation. For example, if the denoising operation is the nth operation of the multi-step denoising operation, the timing vector corresponding to the denoising operation is a vector that can characterize the nth step.
[0038] Specifically, a condition vector is obtained by merging (for example, adding) a reference feature vector and a timing vector, and a second model is used to perform denoising on images that should be denoised based on this condition vector. In addition to the reference feature vector, other feature vectors may also be merged with the timing vector along with the reference feature vector to obtain the condition vector, and this embodiment is not limited to this point. The second model may include multiple network layers, and each of the multiple network layers includes at least one first network layer. The condition vector is fused with latent variables output from each first network layer using a cross-attention mechanism, and the fused vector obtained after fusion is input to the network layer after the first network layer. The first network layer may be any one or more of the multiple network layers. In this embodiment, because the condition vector is fused with latent variables output from the network layer using a cross-attention mechanism, not only is the image generation capability of the second model improved, but the consumption of computational resources is reduced and image generation efficiency is improved.
[0039] On the other hand, super-resolution processing can be performed on the initial image using at least one third model in the following way. First, a second noise image is acquired, the initial image is upsampled, and the image obtained after upsampling and the second noise image are concatenated to obtain the image to be processed. The number of channels in the image to be processed is the sum of the number of channels in the image obtained after upsampling and the second noise image. For example, if the number of channels in the image obtained after upsampling is 3 and the number of channels in the second noise image is 3, then the number of channels in the image to be processed is 6.
[0040] Next, denoising is performed on the image to be processed using at least one third model to obtain a target image. Specifically, first, multi-step denoising is performed on the image to be processed using one third model to obtain an intermediate image, the resolution of which is higher than that of the initial image. Next, the intermediate image is upsampled, and the image obtained after upsampling and the third noise image are concatified. Furthermore, multi-step denoising is performed on the image obtained after concatification using another third model to obtain a target image with a resolution higher than that of the intermediate image.
[0041] However, any third model may include an encoder and a decoder, each of which includes multiple convolutional layers. In the process of noise reduction processing using the third model, it is necessary to perform a downsampling operation on the feature image with the encoder, perform a convolution operation on the feature image with the convolutional layers included in the encoder, and perform an upsampling operation on the feature image with the decoder, and perform a convolution operation on the feature image with the convolutional layers included in the decoder. In the process of processing the feature image with the encoder, the downsampling operation may be performed first, followed by the convolution operation. In the process of processing the feature image with the decoder, the convolution operation may be performed first, followed by the upsampling operation. This reduces the computational load in the convolution operation process, reduces the consumption of computational resources, and improves image generation efficiency.
[0042] Furthermore, any third model can contain multiple residual blocks, allowing for the processing of feature images with relatively high resolution using fewer residual blocks and feature images with relatively low resolution using more residual blocks, thereby reducing the consumption of computational resources.
[0043] In this embodiment, one specific application scenario is when graphic design or advertising requires the use of numerous image assets. The user provides pre-selected or created reference images, and then the system can enrich the available image assets by generating a series of asset images that have similar content, themes, and styles, centered around those reference images.
[0044] Another specific application scenario involves a creator first producing an image work, and then using the solution provided in this disclosure to generate a series of related images similar to the original work in terms of content, theme, and style. The creator can then select some of these images and recreate them, thereby obtaining a better work.
[0045] This embodiment is not limited to the above application scenario and can be applied to other scenarios. The image generation method provided by this disclosure obtains a reference image and a first noise image to obtain a reference feature vector corresponding to the reference image, generates a target image based on the reference feature vector and the first noise image, and the target image and the reference image have high-level features that are partially the same. In this embodiment, the image generation is guided by using a reference image input by the user as an image generation condition, and in order to obtain a target image that has high-level semantic information that is partially the same as the reference image, the image generation effect is improved and a richer image generation method is provided to the user.
[0046] While the above embodiments describe the operation of the methods of the embodiments of this disclosure in a specific order, it should be noted that this does not require or imply that these operations must be performed in that specific order, or that all operations must be performed, in order to achieve the desired results. Instead, the steps shown in the flowchart may be performed in a different order. Furthermore, or instead, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be divided into multiple steps.
[0047] Corresponding to the embodiments of the image generation method described above, this disclosure further provides embodiments of an image generation apparatus.
[0048] As shown in Figure 4, Figure 4 is a block diagram of an image generation apparatus according to one exemplary embodiment of the present disclosure, which may include an acquisition module 401, an extraction module 402, and a processing module 403.
[0049] The acquisition module 401 is used to acquire the reference image and the first noise image.
[0050] The extraction module 402 is used to obtain a reference feature vector corresponding to the reference image.
[0051] The processing module 403 is used to denoise the first noise image based on the reference feature vector and obtain a target image after variation of the reference image. The target image and the reference image have high-level semantic information, with some parts being the same.
[0052] In some embodiments, the extraction module 402 may include an extraction submodule and a concatenation submodule (not shown).
[0053] The extraction submodule is used to obtain multiple different high-level semantic feature vectors corresponding to a reference image using multiple different primary models.
[0054] The Concut submodule is used to perform a concatenation process on multiple different high-level semantic feature vectors to obtain a reference feature vector.
[0055] In other embodiments, the processing module 403 may include a processing submodule and a super-resolution submodule (not shown).
[0056] The processing submodule is used to perform a multi-step denoising operation on the first noisy image using a second model based on a reference feature vector and a first noisy image, thereby obtaining an initial image.
[0057] The super-resolution submodule is used to perform super-resolution processing on the initial image using at least one third model to obtain a target image, the target image having a higher resolution than the initial image.
[0058] In other embodiments, the processing submodule can use the second model to perform a denoising operation at any step of the multistep operation in the following manner: Determine the image to be denoised; if the denoising operation is the first denoising operation, the image to be denoised is the first noisy image; if the denoising operation is not the first denoising operation, the image to be denoised is the result of the denoising operation in the previous step. Determine the timing vector corresponding to the denoising operation, and use the second model to perform denoising on the image to be denoised based on the reference feature vector and the timing vector.
[0059] In other embodiments, the processing submodule utilizes the second model in the following manner to perform denoising on images that should be denoised based on a reference feature vector and a timing vector: A condition vector is obtained based on at least the reference feature vector and the timing vector, and the second model is used to perform denoising on images that should be denoised based on the condition vector.
[0060] In other embodiments, the second model includes multiple network layers, each of which includes at least one first network layer. The processing submodule uses the second model to denoise images that should be denoised based on condition vectors in the following manner: The condition vectors are fused with latent variables output from each first network layer using a cross-attention mechanism, and the resulting fused vector is input to the network layer after the first network layer.
[0061] In other embodiments, the super-resolution submodule is used to acquire a second noise image, upsample the initial image, concatenate it with the second noise image to obtain an image to be processed, and perform noise reduction processing on the image to be processed using at least one third model to obtain a target image.
[0062] In other embodiments, an optional third model includes an encoder and a decoder, each of which includes multiple convolutional layers. The process of denoising using the third model includes performing a downsampling operation before performing a convolution operation in the convolutional layers included in the encoder, and performing an upsampling operation after performing a convolution operation in the convolutional layers included in the decoder.
[0063] Regarding embodiments of the apparatus, they basically correspond to embodiments of the method, so relevant sections should be referred to the partial description of the embodiment of the method. The embodiments of the apparatus described above are merely illustrative, and the units described as separating members may or may not be physically separated, and the members referred to as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Depending on the actual needs, some or all of the units can be selected to achieve the objectives of the solutions of the embodiments of this disclosure. Those skilled in the art can understand and implement them without requiring any creative work.
[0064] Figure 5 is a schematic block diagram of an electronic device according to some embodiments of the present disclosure. As shown in Figure 5, the electronic device 910 includes a processor 911 and a memory 912 and can be used to implement a client or a server. The memory 912 is used to non-temporarily store computer executable instructions (e.g., one or more computer program modules). The processor 911 is used to execute the computer executable instructions, and when the computer executable instructions are executed by the processor 911, one or more steps of the image generation method described above can be executed, thereby implementing the image generation method described above. The memory 912 and the processor 911 may be connected to each other by a bus system and / or other forms of connection mechanisms (not shown).
[0065] For example, the processor 911 may be a central processing unit (CPU), a graphics processing unit (GPU), or another form of processing unit having data processing capability and / or program execution capability. For example, the central processing unit (CPU) may be an X86 or ARM architecture, etc. The processor 911 may be a general-purpose processor or a dedicated processor, and can perform desired functions by controlling other assemblies in the electronic device 910.
[0066] For example, memory 912 may include any combination of one or more computer program products, and computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules can be stored in the computer-readable storage media, and the processor 911 can realize various functions of the electronic device 910 by executing one or more computer program modules. The computer-readable storage media may further store various application programs, various data, and various data used and / or generated by application programs.
[0067] In the embodiments of this disclosure, the specific functions and technical effects of the electronic device 910 can be found in the above description of the image generation method and will not be described again here.
[0068] Figure 6 is a schematic block diagram of another electronic device relating to some embodiments of the present disclosure. The electronic device 920 is suitable, for example, for implementing an image generation method relating to an embodiment of the present disclosure. The electronic device 920 may be a terminal device or the like and can be used to realize a client or server. The electronic device 920 includes, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and wearable electronic devices, as well as fixed terminals such as digital TVs, desktop computers, and smart home devices. Note that the electronic device 920 shown in Figure 6 is merely an example and does not limit the functions and scope of use of the embodiments of the present disclosure.
[0069] As shown in Figure 6, the electronic device 920 may include a processing unit (e.g., a central processor, graphics processor, etc.) 921, which can perform various appropriate operations and processes based on programs stored in read-only memory (ROM) 922 or programs loaded from storage device 928 into random access memory (RAM) 923. RAM 923 further stores various programs and data necessary for the operation of the electronic device 920. The processing unit 921, ROM 922, and RAM 923 are connected to each other via bus 924. An input / output (I / O) interface 925 is also connected to bus 924.
[0070] Typically, input devices 926, including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, and gyroscopes; output devices 927, including, for example, liquid crystal displays (LCDs), speakers, and oscillators; storage devices 928, including, for example, tapes and hard disks; and communication devices 929 can be connected to the I / O interface 925. The communication device 929 can allow the electronic device 920 to exchange data with other electronic devices via wireless or wired communication. Figure 6 illustrates the electronic device 920 with various devices, but it should be understood that it is not necessary to implement or include all of the illustrated devices. Instead, the electronic device 920 may implement or include more or less of the devices.
[0071] For example, according to an embodiment of the present disclosure, the image generation method can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program contained on a non-temporary computer-readable medium, the computer program including program code for executing the image generation method. In such an embodiment, the computer program is downloaded and installed from a network by a communication device 929, or installed from a storage device 928, or installed from a ROM 922. When executed by a processing device 921, the computer program performs functions limited to those in the image generation method according to an embodiment of the present disclosure.
[0072] Figure 7 is a schematic diagram of a storage medium according to some embodiments of the present disclosure. For example, as shown in Figure 7, the storage medium 930 may be a non-temporary computer-readable storage medium for storing non-temporary computer-executable instructions 931. When the non-temporary computer-executable instructions 931 are executed by a processor, they can implement the image generation method described in the embodiments of the present disclosure. For example, when the non-temporary computer-executable instructions 931 are executed by the processor, they can perform one or more steps of the image generation method described above.
[0073] For example, the storage medium 930 can be applied to the electronic device described above, and for example, the storage medium 930 may include memory in the electronic device.
[0074] For example, the storage medium may include a memory card for a smartphone, a storage component for a tablet PC, a hard disk for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.
[0075] For example, the description of the storage medium 930 can be found in the description of memory in the embodiment of the electronic device, and redundant details will not be explained again. The specific functions and technical effects of the storage medium 930 can be found in the above description of the image generation method, and will not be explained again here.
[0076] In the context of this disclosure, a computer-readable medium may be a tangible medium that contains or can store programs used by or in connection with an instruction execution system, apparatus, or device. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. A computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media include, but are not limited to, electrical connections with one or more wires, portable computer magnetic disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium that contains or stores programs used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include data signals propagated in the baseband or as part of a carrier wave, including computer-readable program code. Such propagated data signals may take multiple forms and include, but are not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may further be any computer-readable medium other than a computer-readable storage medium, and such computer-readable signal medium may transmit, propagate, or transmit programs used by or in connection with instruction execution systems, apparatus, or devices. The program code contained in the computer-readable medium may be transmitted by any suitable medium, including, but are not limited to, wires, cables, RF (radio frequency), or any suitable combination thereof.
[0077] A person skilled in the art will readily conceive of other solutions to the disclosure after considering it. This disclosure is intended to cover all variations, uses, or adaptive changes of the disclosure, which, in accordance with the general principles of the disclosure, include common knowledge or general technical means in the art not disclosed herein. The specification and examples are for illustrative purposes only, and the true scope and spirit of the disclosure are limited by the claims.
[0078] To be understood, this disclosure is not limited to the exact structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope of the invention. The scope of this disclosure is limited only to the appended claims.
Claims
1. An image generation method, The steps include obtaining a reference image and a first noise image, A step of obtaining a reference feature vector corresponding to the reference image, comprising: obtaining a plurality of different high-level semantic feature vectors corresponding to the reference image using a plurality of different first models; and performing a concatenation process on the plurality of different high-level semantic feature vectors to obtain the reference feature vector. An image generation method comprising the steps of performing a noise reduction process on the first noise image based on the aforementioned reference feature vector to obtain a target image.
2. The step of performing noise reduction processing on the first noise image based on the aforementioned reference feature vector to obtain a target image is: Based on the aforementioned reference feature vector, a multi-step denoising operation is performed using the second model to obtain an initial image. The method according to claim 1, comprising the step of performing super-resolution processing on the initial image using at least one third model to obtain the target image, wherein the resolution of the target image is higher than the resolution of the initial image.
3. The step of performing a noise reduction operation at any step of a multi-step noise reduction operation using the second model is: A step of determining an image to be denoised, wherein if the denoising operation in any of the steps is the denoising operation in the first step, the image to be denoised is the first noised image, and if the denoising operation in any of the steps is not the denoising operation in the first step, the image to be denoised is the result of the denoising operation in the previous step. The steps include determining a timing vector corresponding to the noise reduction operation in any of the steps, The method according to claim 2, comprising the step of using the second model to perform noise reduction on the image to be denoised based on the reference feature vector and the timing vector.
4. The step of using the second model to perform noise reduction on the image to be denoised based on the reference feature vector and the timing vector is: The steps include obtaining a condition vector based on at least the reference feature vector and the timing vector, The method according to claim 3, further comprising the step of performing noise reduction on the image to be denoised based on the condition vector in the second model.
5. The second model includes multiple network layers, and each of the multiple network layers includes at least one first network layer. In the second model, the step of performing noise reduction on the image to be denoised based on the condition vector is: The method according to claim 4, further comprising the step of fusing the condition vector and the latent variables output from each first network layer using a cross-attention mechanism, and inputting the fusing vector obtained by the fusing to a network layer after each of the first network layers.
6. The step of performing super-resolution processing on the initial image using at least one third model to obtain the target image is: The steps include acquiring a second noise image and The initial image is upsampled and concatenated with the second noise image to obtain an image to be processed. The method according to claim 2, comprising the step of performing a noise reduction process on the image to be processed using at least one third model to obtain the target image.
7. The method according to claim 6, wherein an arbitrary third model includes an encoder and a decoder, each of the encoder and the decoder includes a plurality of convolutional layers, and the step of performing noise reduction processing using the arbitrary third model includes the steps of: performing a downsampling operation before performing a convolution operation in the convolutional layers included in the encoder; and performing an upsampling operation after performing a convolution operation in the convolutional layers included in the decoder.
8. An image generation device, An acquisition module configured to acquire a reference image and a first noise image, An extraction module configured to obtain a reference feature vector corresponding to the aforementioned reference image, specifically, to obtain multiple different high-level semantic feature vectors corresponding to the aforementioned reference image using multiple different first models, and to perform a concatenation process on the multiple different high-level semantic feature vectors to obtain the aforementioned reference feature vector, An image generation apparatus comprising: a processing module configured to perform noise reduction processing on the first noise image based on the aforementioned reference feature vector to obtain a target image; and a processing module configured to perform noise reduction processing on the first noise image based on the aforementioned reference feature vector to obtain a target image.
9. A computer-readable storage medium in which a computer program is stored, wherein when the computer program is executed by a computer, the computer causes the computer to execute the method described in any one of claims 1 to 7.
10. An electronic device comprising memory and a processor, wherein executable code is stored in the memory, and when the processor executes the executable code, the method according to any one of claims 1 to 7 is realized.
11. A computer program wherein, when the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Feature processing method and apparatus, storage medium and program product
CN112868019A
Single-frame image super-resolution processing method based on diffusion model
CN113177882A
Image generation method, and training method and device of image denoising model
CN115222630A
