Image generation method and related equipment

Abstract

This disclosure provides an image generation method which includes extracting prompt information from an acquired base image, inputting the prompt information into a trained diffusion model to obtain a predicted image by a diffusion process, acquiring explanatory features of the predicted image, predicting the resource allocation effect of the predicted image based on the explanatory features, and determining a target image based on the resource allocation effect of the predicted image. Based on the above image generation method, this disclosure further provides an image generation apparatus, electronic equipment, storage medium, and program product.

Description

Technical Field

[0001] (Cross - reference to related applications) This application claims the priority of a Chinese invention patent application titled "Image Generation Method and Related Equipment" with an application number of 2024101310776, which was filed on January 30, 2024. All of the content of this application is incorporated into this application by reference.

[0002] (Technical Field) This disclosure relates to the field of image generation technology, and particularly to image generation methods and related equipment.

Background Art

[0003] In the image production process of the prior art, for images with relatively good effects after resource allocation, these images need to be filtered and new images need to be reprinted based on these images.

[0004] Conventionally, such a reprint process generally depends on manual design and creativity. Team members need to invest a large amount of time to conceive and produce similar high - quality images. In addition, although it is a new idea for other members to obtain the images reprinted by others from other routes, there may also be problems such as unstable content quality.

Summary of the Invention

Problems to be Solved by the Invention

[0005] In view of this, the purpose of this disclosure is to propose an image generation method and related equipment.

Means for Solving the Problems

[0006] This disclosure provides an image generation method which includes extracting prompt information from an acquired base image, inputting the prompt information into a trained diffusion model to obtain a predicted image by a diffusion process, obtaining explanatory features of the predicted image, predicting the resource allocation effect of the predicted image based on the explanatory features, and determining a target image based on the resource allocation effect of the predicted image.

[0007] Based on the image generation method described above, an embodiment of the present disclosure provides an image generation device comprising: an extraction module for extracting prompt information from an acquired base image; a diffusion module for inputting the prompt information into a trained diffusion model and obtaining a prediction waiting image by a diffusion process; a prediction module for acquiring explanatory features of the prediction waiting image and predicting the resource allocation effect of the prediction waiting image based on the explanatory features; and a decision module for determining a target image based on the resource allocation effect of the prediction waiting image.

[0008] Furthermore, embodiments of this disclosure provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, wherein the processor implements the above method when it executes the program.

[0009] Embodiments of the present disclosure further provide a non-temporary computer-readable storage medium in which computer instructions for causing a computer to perform the above method are stored.

[0010] Embodiments of the present disclosure further provide a computer program product that, when run on a computer, includes computer program instructions that cause the computer to perform the above method. [Effects of the Invention]

[0011] The above-described image generation method and related equipment include extracting prompt information from acquired base images, inputting the prompt information into a trained diffusion model to obtain a prediction-wait image through a diffusion process, acquiring explanatory features of the prediction-wait image, predicting the resource allocation effect of the prediction-wait image based on the explanatory features, and determining a target image based on the resource allocation effect of the prediction-wait image. The embodiments of this disclosure extract prompt information from base images, fully decode high-quality historical images, extract the decoded information to obtain prompt information, generate similar images (prediction-wait images) to the base image based on this prompt information, and generate a large number of similar images at once based on the method described above. Therefore, this disclosure ultimately determines a target image that meets the conditions by predicting the resource allocation effect of similar images. [Brief explanation of the drawing]

[0012] To more clearly illustrate the technical concepts in this disclosure or related technologies, the following briefly introduces the drawings that may be used in the examples or related technical descriptions. Obviously, the drawings in the following description are merely examples of the disclosure, and those skilled in the art can obtain other drawings based on these drawings without any creative effort. [Figure 1] The implementation flow of the image generation method described in some embodiments of this disclosure is shown. [Figure 2] A schematic diagram of images described in some embodiments of this disclosure is shown. [Figure 3] A schematic diagram of an image generation apparatus described in some embodiments of this disclosure is shown. [Figure 4] A schematic diagram of the hardware structure of the electronic device described in the embodiments of this disclosure is shown. [Modes for carrying out the invention]

[0013] To make the purpose, technical proposal, and advantages of this disclosure clearer, the disclosure will be described in more detail below with reference to the drawings, along with specific examples.

[0014] Unless otherwise defined, technical or scientific terms used in the embodiments of this disclosure should have the general meaning understood by a person of general skill in the art to which this disclosure belongs. The terms “first,” “second,” and similar terms used in the embodiments of this disclosure do not indicate any order, number, or importance, but are used solely to distinguish different components. Similar terms such as “include” or “incorporate” mean that the element or substance appearing before it includes the enumerated elements or substances and their equivalents appearing after it, but do not exclude other elements or substances. Similar terms such as “connected” or “linked” are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. Terms such as “up,” “down,” “left,” and “right” are used solely to describe relative positional relationships, and these relative positional relationships change accordingly as the absolute position of the object being described changes.

[0015] Before using any of the technical proposals in each embodiment of this disclosure, it is understood that the user will be informed in an appropriate manner of the type of personal information involved, the scope of use, and the usage scenario, and that the user's permission will be obtained.

[0016] For example, when responding to a user's voluntary request, prompt information is sent to the user to clearly prompt the user that it is necessary to obtain and use the user's personal information in order to perform the requested operation. This allows the user to autonomously choose whether or not to provide personal information to software or hardware such as electronic devices, application programs, servers, or storage media that perform the operation of the proposed technical method of this disclosure based on the prompt information.

[0017] As a selective but non-restrictive implementation, a method for sending prompt information to a user in response to receiving a voluntary request from the user may be, for example, a pop-up window, in which case the prompt information may be presented in text format. The pop-up window may also include selection controls for the user to choose whether to "agree" or "disagree" to providing personal information to the electronic device.

[0018] The above notice and user permission acquisition process are general in nature and do not limit the forms in which this disclosure may be implemented. It should be understood that other methods that comply with applicable laws and regulations may also be applicable to the implementation of this disclosure.

[0019] As mentioned above, in conventional image creation processes, it is necessary to filter out images that have a relatively good effect after resource allocation and reprint new images based on these images. Traditionally, such reprint processes generally rely on artificial design and creativity, requiring team members to invest a significant amount of time in conceiving and creating similar, high-quality images. Furthermore, while obtaining images reprinted by other members from other sources is a new approach, it can lead to problems such as inconsistent content quality.

[0020] Therefore, some embodiments of the present disclosure provide an image generation method, which extracts prompt information from the obtained base image, inputs the prompt information into a trained diffusion model, obtains a predicted image through a diffusion process, acquires the descriptive features of the predicted image, predicts the resource allocation effect of the predicted image based on the descriptive features, and determines a target image based on the resource allocation effect of the predicted image. The embodiments of the present disclosure extract prompt information from a base image. After comprehensively interpreting a high-quality historical image and extracting the information after interpretation to obtain prompt information, an image similar to the base image (predicted image) is generated based on this prompt information. Also, a large number of similar images can be generated at once based on the above method. Therefore, the present disclosure predicts the resource allocation effect of similar images to finally determine a target image that meets the conditions. The image generation method of the embodiments of the present disclosure can quickly generate a large number of similar images on the premise of effectively saving human resources, and by predicting the resource allocation effect in advance, the generated similar images can be filtered to obtain the final similar images, thereby optimizing the actual resource allocation effect of similar images.

[0021] FIG. 1 shows the implementation flow of the image generation method described in some embodiments of the present disclosure. As shown in FIG. 1, this method may include the following steps.

[0022] In step 102, extract and obtain prompt information from the obtained base image. In the embodiments of the present disclosure, since the base image is generally an image with a relatively good resource allocation effect, those skilled in the art need to reprint these images to obtain more images with a good resource allocation effect.

[0023] Step 102 may be understood as a deep analysis of the base image, that is, an understanding and insight into the base image. In order to derive high-quality images similar to the base image, the embodiments of the present disclosure not only need to deeply understand the original content, but also need to accurately grasp those important elements that can ensure the resource allocation effect, and ensure that these features can be retained in the newly generated image.

[0024] In the embodiments of the present disclosure, to evaluate the performance of an image, mainly the following aspects need to be considered.

[0025] First, it is the basic characteristics of the image. The basic characteristics of the image may generally include the resolution, size, sharpness, color tone, and the contrast between the theme and the background of the image.

[0026] Since the basic characteristics of the image can determine the overall tone of the image, it is necessary to consider this aspect when evaluating the performance of the image. It can also be said that it is necessary to determine the basic characteristics of the image first, or the basic characteristics of the image may be called constraints. Only in this way can it be guaranteed that the subsequently generated image and the base image have basic similarity.

[0027] Next, it is the main content of the image. The main content of the image relates to the core elements in the image, that is, what the main body is, what activities the main body is performing, and the state that the main body is expressing in the image.

[0028] Furthermore, it is text information. Since the text information included in the image is generally prominent and generally includes the core information of the image, the text information included in the image can generally play a significant decision-making role in the resource allocation effect.

[0029] Finally, there is image layout design, which refers to the position of each element within an image and its corresponding matching relationships. Image layout design must be included in the image considerations because good layout design can significantly increase the overall expressiveness and appeal of an image, and effectively enhance the effectiveness of resource allocation.

[0030] In some embodiments of this disclosure, obtaining prompt information from an acquired base image includes obtaining base information from the acquired base image, identifying the image content of the base image to obtain content information, and performing information extraction on the base information and the content information to obtain the prompt information.

[0031] In some embodiments of the present disclosure, the basic information includes at least one of the image size, resolution, color tone, and contrast of the basic image.

[0032] Figure 2 shows schematic diagrams of images described in some embodiments of the present disclosure. You can refer to Figure 2, for example, image 1 is circular, images 2 and 3 are rectangular, and images 2 and 3 are of equal size.

[0033] In the embodiments of this disclosure, the aforementioned basic information is the basic characteristics of the picture described above. The basic characteristics may be acquired in the form of direct acquisition at the time of acquisition, or the relevant information may be acquired by a simple computer vision method. Computer vision is the science of how to "show" to a machine, and more specifically, it refers to using a camera and a computer to perform machine vision such as identification, tracking and measurement of a target in place of the human eye, and further performing graphics processing to make the computer processing into an image that is more suitable for distribution to human observation or equipment detection.

[0034] In some embodiments of the present disclosure, obtaining content information by identifying the image content of the base image includes identifying the text content of the base image and obtaining text information, obtaining the position information of the text information and obtaining text position information, identifying the image content of the base image and obtaining image information, identifying the image layout of the base image and obtaining layout information, and obtaining content information based on the text information, the text position information, the image information, and the layout information.

[0035] As shown in Figure 2, in embodiments of the present disclosure, identifying the image content of a base image may include identifying the text content of characters, identifying the position of characters, identifying the content of the image, and further including identifying the layout of the image.

[0036] Identifying the text content of the base image may be done first using neural network technology, and more specifically, it may include the need to first detect where the text is located. In embodiments of this disclosure, text may be detected using an object detection method.

[0037] The task of target detection is to find all targets (objects) of interest in an image. Unlike classification and regression problems, target detection also requires determining the position (location) of the targets in the image and determining the type and position (classification and location) of the identified targets.

[0038] For example, in Figure 2, after identifying the characters, the result is obtained that the characters are located in the upper left corner of the picture.

[0039] Furthermore, after identifying the position of the characters, the detected character position region may be identified to obtain the specific content of the characters.

[0040] In embodiments of this disclosure, the specific content of characters can be identified using Optical Character Recognition (OCR) technology. Optical Character Recognition is a process that performs analysis and identification on an image file of text material to obtain character and layout information. That is, it identifies characters in an image and returns them in text format.

[0041] In embodiments of this disclosure, image information may be obtained by identifying the image content of a base image. This image information may be very specific. For example, it may be possible to identify that the image currently contains perfume of brand A, specifically how many bottles there are, or that it further contains skincare lotion of brand A, specifically how many bottles there are.

[0042] Subsequently, the image layout of the base image may be identified. The image layout here may include multiple image stitching material, single image material, top-of-text presentation, middle-of-text presentation, etc., or refer to Figure 2. For example, the image is multiple image stitching material, with no overlap between each image, image 1 is circular, located on the left side of the entire image, located to the left of image 2, image 2 and image 3 are vertically aligned, image 3 is located below image 2, and both image 2 and image 3 are located on the right side of the image.

[0043] Subsequently, content information is obtained based on the character information, character position information, image information, and layout information. In the embodiments of this disclosure, after obtaining the character information, character position information, image information, and layout information, the aforementioned information can be combined to obtain content information for the image, that is, information that describes the image content, layout, etc., in detail.

[0044] In the embodiments of this disclosure, multimodal technology can also directly generate detailed descriptive information corresponding to an image, for example, a man standing on a surfboard surfing on the sea. It can directly describe the main information of the image in detail.

[0045] In some embodiments of the present disclosure, obtaining prompt information by extracting information from the basic information and the content information includes obtaining the basic information, the character position information and the layout information, extracting core information from the character information and the image information, and obtaining prompt information based on the basic information, the character information, the layout information and the core information.

[0046] In the embodiments of this disclosure, after obtaining character information, character position information, image information, and layout information, combining them allows for obtaining detailed information corresponding to this image. When reprinting the image subsequently, it is reprinted based on prompt information. However, after obtaining the aforementioned detailed information, it is not possible to directly use the detailed information as prompt information. For the embodiments of this disclosure, it is necessary to create a similar image, which only needs to retain the most core content of the base image. Retaining too much detail would, conversely, be detrimental to the subsequent reprinting of the image.

[0047] Therefore, it is necessary to extract content from the detailed information obtained as described above to obtain the corresponding prompt information. The specific steps may include storing the basic information, character position information, and layout information obtained as described above. For example, the basic information may be that image 1 is circular, image 2 is rectangular, image 3 is rectangular, the background color of the picture is warm black, and there is contrast between it and the cosmetic picture.

[0048] For text and image information, it is sufficient to retain their core content. For example, referring to Figure 2, the text information could be "A brand cosmetics are on sale," and it could be extracted as "Cosmetics are on sale." For image information, the aforementioned acquired image information is "The image is made by stitching together three pictures: Image 1 shows a girl holding A brand perfume, Image 2 shows a girl holding A brand skincare lotion, and Image 3 shows a girl holding A brand foundation."

[0049] The final prompt information to be retained may be: "The image is made by stitching together three images, each showing a person with different cosmetics, which may be skincare products or perfumes, a one-paragraph text description indicating that these cosmetics are a good deal should be displayed in the upper left position of the picture, the background color of the entire image should be a warm black tone, and there should be contrast with the cosmetic pictures."

[0050] After the extraction of prompt information from the image is completed in the aforementioned step, it is necessary to generate similar images (prediction pending images) based on the prompt information formed above.

[0051] In step 104, in some embodiments of the present disclosure, obtaining a pending prediction image by a diffusion process includes obtaining a random noise image and gradually converting the random noise image into a continuous image based on the prompt information to obtain the pending prediction image.

[0052] In embodiments of this disclosure, after obtaining high-quality prompt information based on the aforementioned understanding and insight into the image, an image generation algorithm can be used to generate a corresponding similar image based on the original obtained prompt information.

[0053] Specifically, the embodiments of this disclosure employ a stable diffusion algorithm to generate images. The stable diffusion algorithm is a novel image generation technique aimed at generating high-fidelity images in a controllable and stable manner. Based on the concept of image noise reduction, this algorithm gradually transforms random noise patterns into a continuous image structure through a diffusion process.

[0054] The core of the Stable Diffusion algorithm lies in its two main steps.

[0055] The first step is diffusion, which simulates the effect of the natural diffusion process in a high-dimensional data space after inputting the original image, continuously adding noise to the original information and introducing randomness. The second step is dediffusion, which uses a deep learning model to guide the diffusion process to the desired image input, ultimately obtaining the image that needs to be generated. For the denoising process, in addition to including information after adding noise to the original image, it further includes conditional information in each denoising process, and this conditional information is generally described by several letters or consists of several constraints.

[0056] In some embodiments of the present disclosure, obtaining the predicted waiting image by gradually converting the random noise image into a continuous image based on the prompt information includes performing noise reduction on the random noise image using the prompt information as conditional information for the random noise image to obtain the predicted waiting image.

[0057] In the embodiments of this disclosure, the condition information may be prompt information.

[0058] In some embodiments of this disclosure, a trained diffusion model is obtained by training it using a method that involves acquiring images from history images in which a predetermined parameter is higher than a predetermined threshold to obtain a history base image, extracting history prompt information based on the history base image, inputting the history prompt information into an untrained diffusion model to obtain a history prediction waiting image, and training the untrained diffusion model based on the history prediction waiting image and the history base image to obtain the trained diffusion model.

[0059] In the embodiments of this disclosure, the training data for a standard stable diffusion model is often a few generic pictures, which do not match the scenarios of this disclosure. Furthermore, the images generated in this disclosure must resemble the underlying images and have relatively good resource allocation efficiency. Because the requirements for images in this disclosure are specific, images generated by a standard stable diffusion model cannot be used directly.

[0060] In the embodiments of this disclosure, there are many high-quality base images, and the resource allocation effect of these high-quality base images is good. Therefore, the stable diffusion model can be retrained using these high-quality base images to improve the quality of the generated images. Of course, during training, the model may be trained from scratch, or a pre-trained model may be retrained, thus effectively saving the model training time.

[0061] In the training process, the prompt information for high-quality base images may be generated using the aforementioned step for generating prompt information, and then, after generating images based on this prompt information, the associated loss can be calculated.

[0062] Furthermore, since resources are limited after image generation, it is necessary to allocate resources to the best quality image. Therefore, the process does not end after image generation; instead, a model can be set up to predict the resource allocation effect of the generated images, and the image with a relatively good resource allocation effect can be used as the final target image.

[0063] In step 106, in some embodiments of the present disclosure, obtaining descriptive features of the pending image and predicting the resource allocation effect of the pending image based on the descriptive features includes obtaining the descriptive features and type features of the pending image, obtaining the historical resource allocation effect of the base image, obtaining prompt information corresponding to the pending image and obtaining descriptive features of the prompt information, and predicting and obtaining the resource allocation effect of the pending image based on the descriptive features and type features of the pending image, the historical resource allocation effect of the base image, and the descriptive features of the prompt information.

[0064] In some embodiments of the present disclosure, determining a target image based on the resource allocation effect of the predicted image includes determining the predicted image as the target image in response to the resource allocation effect of the predicted image being higher than a preset threshold.

[0065] In the embodiments of this disclosure, a large number of historical images exist, and there is an effect of image information and corresponding image resource allocation, so the model can be trained based on this historical information.

[0066] After the model training is complete, the resource allocation effect may be predicted for the previously acquired images awaiting prediction. In the specific prediction process, explanatory features and type features of the images awaiting prediction may be obtained first. An explanatory feature refers to a low-dimensional vector that represents a single object, which may be a single word, a single product, or a single movie. The property of this vector is that vectors that are close in distance can be given similar meanings to objects that correspond to them.

[0067] Subsequently, explanatory features of the prompt information may be obtained. Also, given that the image awaiting prediction is a reprint of the base image, the type of the image awaiting prediction may be obtained, and of course, the base image matches the type of the image awaiting prediction.

[0068] Subsequently, dense features such as the size, resolution, and historical resource allocation effect of the base image may be acquired.

[0069] Subsequently, the previously acquired features are input into a neural network using a multi-task structure to compute features. Finally, one branch is used for estimation for each target, and the underlying features of each subtask are shared. In this way, parameter sharing effectively reduces computational complexity and increases computation speed.

[0070] The above image generation method extracts prompt information from acquired base images, inputs the prompt information into a trained diffusion model, obtains a prediction-waiting image through a diffusion process, obtains explanatory features of the prediction-waiting image, predicts the resource allocation effect of the prediction-waiting image based on the explanatory features, and determines a target image based on the resource allocation effect of the prediction-waiting image. The embodiment of this disclosure extracts prompt information from base images, fully decodes high-quality historical images, extracts the decoded information to obtain prompt information, generates similar images (prediction-waiting images) to the base image based on this prompt information, and can generate a large number of similar images at once based on the method described above. Therefore, this disclosure ultimately determines a target image that meets the conditions by predicting the resource allocation effect of similar images. The image generation method of the embodiment of this disclosure can quickly generate a large number of similar images while effectively saving human resources, and by predicting the resource allocation effect in advance, it is possible to filter the generated similar images to obtain the final similar image, thereby optimizing the actual resource allocation effect of similar images.

[0071] The methods of the embodiments of this disclosure may be performed by a single device, such as a single computer or server. The methods of the embodiments may also be used in a distributed scenario in which multiple devices cooperate to complete the process. In such a distributed scenario, one of these multiple devices may perform only one or more steps of the methods of the embodiments of this disclosure, and these multiple devices may interact with each other to complete the method.

[0072] Several embodiments of this disclosure have been described above. Other embodiments are within the scope of the appended claims. In some cases, the operations or steps described in the claims may be performed in a different order than those in the embodiments above, and the desired results can still be achieved. Furthermore, the processes depicted in the drawings do not necessarily require a specific order or sequence shown to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0073] Based on the same inventive concept and corresponding to the methods of any of the above embodiments, the present disclosure further provides an image generation apparatus comprising: an extraction module 302 for extracting prompt information from an acquired base image; a diffusion module 304 for inputting the prompt information into a trained diffusion model and obtaining a predicted waiting image by a diffusion process; a prediction module 306 for acquiring explanatory features of the predicted waiting image and predicting the resource allocation effect of the predicted waiting image based on the explanatory features; and a determination module 308 for determining a target image based on the resource allocation effect of the predicted waiting image.

[0074] In some embodiments of the present disclosure, the extraction module 302 comprises an extraction unit for extracting basic information from the acquired basic image, an identification unit for identifying the image content of the basic image and obtaining content information, and an information extraction unit for performing information extraction on the basic information and the content information to obtain prompt information.

[0075] In some embodiments of the present disclosure, the basic information includes at least one of the image size, resolution, color tone, and contrast of the basic image.

[0076] In some embodiments of the present disclosure, the identification unit includes identifying the text content of the base image to obtain text information, obtaining the location information of the text information to obtain text location information, identifying the image content of the base image to obtain image information, identifying the image layout of the base image to obtain layout information, and obtaining content information based on the text information, the text location information, the image information, and the layout information.

[0077] In some embodiments of the present disclosure, the extraction unit includes acquiring the basic information, the character position information and the layout information, extracting core information of the character information and the image information, and obtaining the prompt information based on the basic information, the character information, the layout information and the core information.

[0078] In some embodiments of the present disclosure, the diffusion module 304 comprises a first acquisition unit for acquiring a random noise image and a conversion unit for gradually converting the random noise image into a continuous image based on the prompt information to obtain the predicted waiting image.

[0079] In some embodiments of the present disclosure, the conversion unit includes using the prompt information as conditional information for the random noise image to denoise the random noise image to obtain the predicted waiting image.

[0080] Some embodiments of the present disclosure further include: a history image acquisition module for obtaining a history base image by acquiring images from history images in which a predetermined parameter is higher than a predetermined threshold; a history prompt information extraction module for obtaining history prompt information based on the history base image; an input module for obtaining a history prediction waiting image by inputting the history prompt information to an untrained diffusion model; and a training module for training the untrained diffusion model based on the history prediction waiting image and the history base image to obtain a trained diffusion model.

[0081] In some embodiments of the present disclosure, the prediction module 306 includes a second acquisition unit for acquiring descriptive features and type features of the prediction-awaited image; a third acquisition unit for acquiring the history resource allocation effect of the base image; a fourth acquisition unit for acquiring prompt information corresponding to the prediction-awaited image and acquiring descriptive features of the prompt information; and a prediction unit for predicting and obtaining the resource allocation effect of the prediction-awaited image based on the descriptive features and type features of the prediction-awaited image, the history resource allocation effect of the base image, and the descriptive features of the prompt information.

[0082] In some embodiments of the present disclosure, the decision module 308 includes a decision unit for determining the predicted image as the target image in response to the resource allocation effect of the predicted image being higher than a preset threshold.

[0083] For the sake of clarity, the above apparatus will be described by dividing it into various modules according to their function. Of course, when implementing this disclosure, the functions of each module may be implemented in the same or multiple software and / or hardware. The apparatus of the above embodiment is used to implement the corresponding image generation method in any one of the embodiments described above, and has the beneficial effects of the embodiment of the corresponding method, and will not be described further here.

[0084] Based on the same inventive concept and corresponding to the methods of any of the above embodiments, the present disclosure further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, wherein the processor, upon execution of the program, implements the image generation method described in any one of the above embodiments.

[0085] Figure 4 shows a schematic diagram of the hardware structure of a more specific electronic device according to this embodiment, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. Here, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 communicate with each other via the bus 1050.

[0086] The processor 1010 may be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits, for executing the associated program and realizing the technical proposal according to the embodiments herein.

[0087] Memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage, dynamic storage, etc. Memory 1020 can store the operating system and other application programs, and when the technical proposal according to the embodiments herein is implemented by software or firmware, the relevant program code is stored in memory 1020 and called and executed by processor 1010.

[0088] The input / output interface 1030 is used to connect input / output modules to enable the input and output of information. The input / output modules may be configured as components in the device (not shown) or may be connected externally to the device to provide the corresponding functions. Here, input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0089] The communication interface 1040 is used to connect a communication module (not shown) to enable communication interaction between this device and other devices. Here, the communication module may implement communication using a wired method (e.g., USB, network cable, etc.) or a wireless method (e.g., mobile network, Wi-Fi, Bluetooth®, etc.).

[0090] Bus 1050 includes paths for transmitting information between the various components of the device (e.g., processor 1010, memory 1020, input / output interface 1030, and communication interface 1040).

[0091] Although the above-mentioned equipment only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in a specific implementation process, this equipment may further include other components necessary to achieve normal operation. Furthermore, those skilled in the art will understand that the above-mentioned equipment may include only the components necessary to implement the embodiment described herein, and does not necessarily need to include all the components shown in the figures.

[0092] The electronic device of the above embodiment is used to implement the corresponding image generation method in any one of the embodiments described above, and has the beneficial effects of the embodiment of the corresponding method, which will not be described further here.

[0093] Based on the same inventive concept and corresponding to the method of any of the above embodiments, the present disclosure further provides a non-temporary computer-readable storage medium which stores computer instructions for causing the computer to execute the image generation method described in any one of the above embodiments.

[0094] The computer-readable media of this embodiment include persistent and non-persistent, removable and non-removable media, and information can be stored by any method or technique. The information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only disk memory (CD-ROM), digital multipurpose disk (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that may be used to store information accessible by computing equipment.

[0095] The computer instructions stored in the storage medium of the above embodiment are used to cause the computer to execute the image generation method described in any one of the above embodiments, and have the beneficial effects of the embodiment of the corresponding method, which will not be described further here.

[0096] Based on the same inventive concept, and corresponding to the image generation method described in any of the above embodiments, the present disclosure further provides a computer program product including computer program instructions. In some embodiments, the computer program instructions may be executed by one or more processors of the computer to cause the computer and / or the processor to execute the image generation method. Corresponding to the execution body corresponding to each step in each embodiment of the image generation method, the processor that executes the corresponding step may belong to the corresponding execution body.

[0097] The computer program product of the above embodiment is used to cause the computer and / or the processor to execute the image generation method described in any one of the above embodiments, and has the beneficial effects of the embodiment of the applicable method, which will not be described further here.

[0098] Those skilled in the art will understand that the consideration of the above-described examples is illustrative and not intended to imply that the scope of the Disclosure (including the claims) is limited to these examples, that in the spirit of the Disclosure, technical features in the above-described examples or different examples can be combined, that the steps can be carried out in any order, and that many other variations of different embodiments of the embodiments of the Disclosure described above exist, which are not provided in detail for the sake of simplicity.

[0099] Furthermore, in order to simplify the explanation and discussion, and to avoid making it difficult to understand the embodiments of this disclosure, the provided drawings may or may not show known power / ground connections of integrated circuit (IC) chips and other components. Also, in order to avoid making it difficult to understand the embodiments of this disclosure, the apparatus may be shown in the form of a block diagram, taking into consideration the following fact: the details of the embodiments of the apparatus in these block diagrams depend heavily on the platform on which the embodiments of this disclosure are carried out (i.e., these details should be fully within the understanding of those skilled in the art). Where specific details (e.g., circuits) are described to illustrate exemplary embodiments of this disclosure, it will be obvious to those skilled in the art that the embodiments of this disclosure can be carried out even if these specific details are absent or changed. Therefore, these descriptions should be considered explanatory, not restrictive.

[0100] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications and variations of these embodiments will be apparent to those skilled in the art as described herein. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used in the embodiments discussed.

[0101] The embodiments of this disclosure are intended to include all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. This involves extracting prompt information from the acquired base image, The prompt information is input to a trained diffusion model, and a prediction image is obtained through the diffusion process. The process involves obtaining explanatory features of the image awaiting prediction, and predicting the resource allocation effect of the image awaiting prediction based on these explanatory features. This includes determining a target image based on the resource allocation effect of the aforementioned prediction awaiting image, Image generation method.

2. Extracting prompt information from the acquired base image is possible. Extracting basic information from the acquired basic image, To identify the image content of the aforementioned base image and obtain content information, The method according to claim 1, comprising extracting information from the basic information and the content information to obtain the prompt information.

3. The method according to claim 2, wherein the basic information includes at least one of the image size, resolution, color tone, and contrast of the basic image.

4. Identifying the image content of the aforementioned base image and obtaining content information is, To identify the text content of the aforementioned base image and obtain text information, To obtain the position information of the aforementioned character information, To identify the image content of the aforementioned base image and obtain image information, Identifying the image layout of the aforementioned base image and obtaining layout information, The method according to claim 2, comprising obtaining content information based on the character information, the character position information, the image information, and the layout information.

5. To obtain the prompt information by extracting information from the aforementioned basic information and the aforementioned content information, To obtain the aforementioned basic information, the aforementioned character position information and the aforementioned layout information, Extracting core information from the aforementioned text information and image information, The method according to claim 4, comprising obtaining the prompt information based on the basic information, the character information, the layout information, and the core information.

6. Obtaining a prediction image through a diffusion process is possible. To acquire random noise images, The method according to claim 1, comprising gradually converting the random noise image into a continuous image based on the prompt information to obtain the predicted waiting image.

7. Based on the prompt information, gradually converting the random noise image into a continuous image to obtain the predicted waiting image is: The method according to claim 6, comprising using the prompt information as condition information for the random noise image, performing noise reduction on the random noise image to obtain the predicted waiting image.

8. A method for obtaining a historical base image by acquiring images from historical images where pre-set parameters are higher than a pre-set threshold, and A method for extracting and obtaining history prompt information based on the aforementioned history base image, A method for obtaining a history prediction waiting image by inputting the history prompt information into an untrained diffusion model, The method according to claim 1, wherein the trained diffusion model is obtained by training the untrained diffusion model based on the history prediction waiting image and the history base image, and thereby obtaining the trained diffusion model.

9. Obtaining explanatory features of the image awaiting prediction and predicting the resource allocation effect of the image awaiting prediction based on these explanatory features is: To obtain the explanatory and type features of the aforementioned image awaiting prediction, To obtain the historical resource allocation effect of the aforementioned base image, The process involves obtaining prompt information corresponding to the aforementioned prediction waiting image, and obtaining explanatory features of the prompt information. The method according to claim 1, comprising predicting and obtaining the resource allocation effect of the predicted waiting image based on the descriptive features and type features of the predicted waiting image, the historical resource allocation effect of the base image, and the descriptive features of the prompt information.

10. Determining the target image based on the resource allocation effect of the aforementioned prediction await image is The method according to claim 1, comprising determining the predicted image as the target image in response to the resource allocation effect of the predicted image being higher than a preset threshold.

11. An extraction module for extracting prompt information from acquired base images, A diffusion module for inputting the prompt information into a trained diffusion model and obtaining a predicted image through a diffusion process, A prediction module for obtaining explanatory features of the image awaiting prediction and predicting the resource allocation effect of the image awaiting prediction based on the explanatory features, The system includes a decision module for determining a target image based on the resource allocation effect of the predicted waiting image. Image generation device.

12. The extraction module is An extraction unit for extracting basic information from the acquired basic image, An identification unit for identifying the image content of the aforementioned base image and obtaining content information, The apparatus according to claim 11, further comprising an information extraction unit for performing information extraction on the basic information and the content information to obtain the prompt information.

13. The apparatus according to claim 12, wherein the basic information includes at least one of the image size, resolution, color tone, and contrast of the basic image.

14. The aforementioned identification unit is To identify the text content of the aforementioned base image and obtain text information, To obtain the position information of the aforementioned character information, To identify the image content of the aforementioned base image and obtain image information, Identifying the image layout of the aforementioned base image and obtaining layout information, The apparatus according to claim 12, comprising obtaining content information based on the character information, the character position information, the image information, and the layout information.

15. The extraction unit is To obtain the aforementioned basic information, the aforementioned character position information and the aforementioned layout information, Extracting core information from the aforementioned text information and image information, The apparatus according to claim 14, comprising obtaining the prompt information based on the basic information, the character information, the layout information, and the core information.

16. The aforementioned diffusion module is A first acquisition unit for acquiring random noise images, The apparatus according to claim 11, further comprising a conversion unit for gradually converting the random noise image into a continuous image based on the prompt information to obtain the predicted waiting image.

17. The aforementioned conversion unit is The apparatus according to claim 16, further comprising using the prompt information as conditional information for the random noise image, performing noise reduction on the random noise image to obtain the predicted waiting image.

18. A history image acquisition module for obtaining a history base image by acquiring images from history images where the pre-set parameters are higher than a pre-set threshold, A history prompt information extraction module for extracting history prompt information based on the aforementioned history base image, An input module for obtaining a history prediction waiting image by inputting the history prompt information into an untrained diffusion model, The apparatus according to claim 11, further comprising a training module for training the untrained diffusion model based on the history prediction waiting image and the history base image to obtain the trained diffusion model.

19. The aforementioned prediction module is A second acquisition unit for acquiring explanatory features and type features of the aforementioned image awaiting prediction, A third acquisition unit for obtaining the historical resource allocation effect of the aforementioned base image, A fourth acquisition unit for acquiring prompt information corresponding to the prediction waiting image and for acquiring explanatory features of the prompt information, The apparatus according to claim 11, further comprising a prediction unit for predicting and obtaining the resource allocation effect of the prediction-waiting image based on the descriptive features and type features of the prediction-waiting image, the historical resource allocation effect of the base image, and the descriptive features of the prompt information.

20. The aforementioned decision module is The apparatus according to claim 11, further comprising a determination unit for determining the predicted image as the target image in response to the resource allocation effect of the predicted image being higher than a preset threshold.

21. An electronic device comprising memory, a processor, and a computer program stored in memory and operable on the processor, wherein the processor, upon execution of the program, realizes the method according to any one of claims 1 to 10. electronic equipment.

22. A computer instruction is stored which causes the computer to perform the method described in any one of claims 1 to 10. A non-temporary computer-readable storage medium.

23. When run on a computer, the computer program includes instructions that cause the computer to perform the method described in any one of claims 1 to 10. Computer program products.

Images