Image generation method and apparatus, device, storage medium, and program product

Abstract

The present application relates to the technical field of computer vision, and in particular to an image generation method and apparatus, a device, a storage medium, and a program product. The method comprises: performing a first segmentation operation on a first image to obtain a subject image, the subject image being an image corresponding to a specified object in the first image; performing an optimization operation on the subject image to obtain an optimized subject image, the optimization operation being used for eliminating an image defect in the subject image which is introduced by the segmentation operation; and generating a target image on the basis of the optimized subject image. In the solution above, image quality optimization is performed on a subject image having undergone a segmentation operation, and a target image is generated on the basis of the subject image having undergone the image quality optimization, such that the quality of image generation can be effectively improved.

Description

Image generation method, device, equipment, storage medium and program product TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer vision, and particularly relate to an image generation method, device, equipment, storage medium and program product. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, the application of machine learning models is also becoming more and more widespread. In particular, in the field of image processing, a poster can be generated based on an image through a machine learning model.

[0003] In related technologies, the process of generating a poster from an image through a machine learning model can be as follows: a user uploads an image with a product subject to the machine learning model, the machine learning model extracts the product subject from the image, and then generates an image containing the product subject as a poster based on the product subject.

[0004] However, in the scheme shown in the above related technology, since the image uploaded by the user with the product subject has some unclear images or complex images, it is easy to cause the image generated by the machine learning model to have poor quality, and the effect of poster generation is poor.

[0005] SUMMARY

[0006] Embodiments of the present application provide an image generation method, device, equipment, storage medium and program product, which can improve the quality of image generation. The technical scheme is as follows:

[0007] On the one hand, an image generation method is provided, and the method comprises:

[0008] performing a first segmentation operation on a first image to obtain a subject image; the subject image is an image corresponding to a specified object in the first image;

[0009] performing an optimization operation on the subject image to obtain an optimized subject image; the optimization operation is used to eliminate image defects introduced by the segmentation operation in the subject image;

[0010] generating a target image based on the optimized subject image; the target image is a poster with the specified object as the subject.

[0011] On the other hand, an image generation device is provided, and the device comprises:

[0012] a segmentation operation execution module configured to perform a first segmentation operation on a first image to obtain a subject image; the subject image is an image corresponding to a specified object in the first image;

[0013] an optimization operation execution module configured to perform an optimization operation on the subject image to obtain an optimized subject image, the optimization operation being configured to eliminate image defects introduced by the segmentation operation in the subject image;

[0014] a target image generation module configured to generate a target image based on the optimized subject image, the target image being a poster with the specified object as a subject.

[0015] In some embodiments, the optimization operation comprises at least one of a first optimization operation and a second optimization operation:

[0016] the first optimization operation is configured to eliminate holes and discrete points in the subject image;

[0017] the second optimization operation is configured to eliminate edge segmentation jaggies in the subject image.

[0018] In some embodiments, in a case where the optimization operation comprises the first optimization operation, the optimization operation execution module is configured to perform a close operation on a subject region in a segmentation mask of the subject image, the segmentation mask comprising one or more subject regions and one or more non-subject regions, the subject region being a continuous region corresponding to the specified object in the first image, the non-subject region being a continuous region other than the subject region in the first image;

[0019] for a first subject region in the segmentation mask of the subject image, in a case where an area of the first subject region is less than a first area, the first subject region being one of the one or more subject regions, the first subject region is removed;

[0020] for a first non-subject region in the segmentation mask of the subject image, in a case where the first non-subject region is located inside any of the subject regions, the first non-subject region being one of the one or more non-subject regions, a fill operation is performed on the first non-subject region.

[0021] In some embodiments, in a case where the optimization operation comprises the second optimization operation, the optimization operation execution module is configured to map segmentation probability values of pixels in the first image to a first range interval, the segmentation probability values being obtained by the first segmentation operation, the pixels belonging to the subject image;

[0022] map segmentation probability values of pixels in a non-edge region of the subject image to a second range interval, the non-edge region being a region other than an edge region in the subject image, the second range interval being located inside the first range interval.

[0023] In some embodiments, the apparatus further includes:

[0024] a segmentation image obtaining module configured to perform a second segmentation operation on the target image to obtain a segmentation image, the segmentation image containing one or more element instances;

[0025] an editing module configured to perform an editing operation on the element instance to obtain a first edited image;

[0026] The editing operation includes, but is not limited to, any one of the following operations: zoom-in operation, zoom-out operation, and moving operation.

[0027] In some embodiments, the apparatus further includes:

[0028] a feature code obtaining module configured to obtain a first feature code and a second feature code;

[0029] an output feature code obtaining module configured to input the first feature code and the second feature code into a diffusion model to obtain an output feature code of the diffusion model, the output feature code being a feature code obtained by decoding after performing a dot product operation on the first feature code and the second feature code;

[0030] a second edited image obtaining module configured to input the output feature code into a VAE (Variational Autoencoder) model to obtain a second edited image output by the VAE model;

[0031] The first feature code is a feature code output by a control unit, the control unit is configured to perform an image quality processing operation on the first edited image, the image quality processing operation includes retaining feature details of the first edited image in the second edited image and maintaining color consistency between the second edited image and the first edited image, the second feature code is a feature code obtained by splicing the first feature code and a down-sampling mask and adding noise, and the down-sampling mask is used to indicate a hole region in the first edited image.

[0032] In some embodiments, the apparatus further includes:

[0033] a text layout information determining module configured to determine text layout information according to the target image, the text layout information being used to indicate a type of target text and a position of the target text in the target image;

[0034] a target text adding module configured to add the target text in the target image according to the text layout information.

[0035] In some embodiments, the target image generation module is configured to input the optimized subject image into a deconstruction model to obtain an output of the deconstruction model as a reference prompt word; obtain a background description text according to the reference prompt word; and generate the target image according to the background description text, wherein a background in the target image matches the background description text.

[0036] In some embodiments, the target image generation module is configured to display the reference prompt word to a user; and receive the background description text input by the user.

[0037] In some embodiments, the target image generation module is configured to input the reference prompt word into a generative language model to obtain the background description text output by the generative language model.

[0038] In some embodiments, the target image generation module is configured to determine one or more candidate background templates matching the optimized subject image from background templates; randomly determine a background image from the one or more candidate background templates; and generate the target image according to the background image and the optimized subject image.

[0039] In another aspect, a computer device is provided, which includes a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the image generation method as described above.

[0040] In another aspect, a computer readable storage medium is provided, which stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by a processor to implement the image generation method as described above.

[0041] In yet another aspect, a computer program product is provided, which includes a computer program stored in a computer readable storage medium. A processor of a computer device reads the computer program from the computer readable storage medium, and the processor executes the computer program to enable the computer device to perform the image generation method provided in the various optional implementation manners described above.

[0042] The technical solutions provided in the present application can include the following beneficial effects:

[0043] The computer device performs a segmentation operation on the image corresponding to the specified object in the first image to extract a subject image from the first image. Since the subject image has an image quality problem when being segmented, the computer device performs an optimization operation on the subject image to eliminate the image quality problem introduced in the segmentation operation. The image quality of the optimized subject image obtained after the optimization operation is better, and the specified object can be displayed more clearly. When generating a target image (i.e., a poster containing the specified object) based on the optimized subject image whose image quality problem has been eliminated, the specified object in the optimized subject image is clearer, so that the quality of the generated target image is better, and the generation effect of the poster image is effectively improved.

[0044] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0045] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] FIG. 1 is a system configuration diagram of an image generation method according to an embodiment of the present application;

[0047] FIG. 2 is a flowchart of an image generation method according to an embodiment of the present application;

[0048] FIG. 3 is a flowchart of an image generation method according to an embodiment of the present application;

[0049] FIG. 4 is a flowchart of an image generation method according to an embodiment of the present application;

[0050] FIG. 5 is a flowchart of an image generation method according to an embodiment of the present application;

[0051] FIG. 6 is a flowchart of an image generation method according to an embodiment of the present application;

[0052] FIG. 7 is a flowchart of an image generation method according to an embodiment of the present application;

[0053] FIG. 8 is a flowchart of an image generation method according to an embodiment of the present application;

[0054] FIG. 9 is a flowchart of an AI poster generation method according to an exemplary embodiment of the present application;

[0055] FIG. 10 is a schematic diagram of a subject segmentation of a commodity according to an exemplary embodiment of the present application;

[0056] Fig. 11 is a schematic diagram of hole segmentation and small discrete point optimization for an image according to one example embodiment of the present application;

[0057] Fig. 12 is a schematic diagram of sawtooth segmentation optimization according to one example embodiment of the present application;

[0058] Fig. 13 is a schematic diagram of edge condition control according to one example embodiment of the present application;

[0059] Fig. 14 is a schematic diagram of random background generation according to one example embodiment of the present application;

[0060] Fig. 15 is a schematic diagram of SAM segmentation of all elements according to one example embodiment of the present application;

[0061] Fig. 16 is a schematic diagram of secondary editing-supplement generation according to one example embodiment of the present application;

[0062] Fig. 17 is a schematic diagram of a comparison between depth detection and saliency detection according to one example embodiment of the present application;

[0063] Fig. 18 is a schematic diagram of a network generator structure according to one example embodiment of the present application;

[0064] Fig. 19 is a block diagram of an image generation apparatus according to one example embodiment of the present application;

[0065] Fig. 20 is a schematic diagram of a computer device according to one example embodiment of the present application. DETAILED DESCRIPTION

[0066] The example embodiments will be described in detail herein with reference to the attached drawings. The following description is made with reference to the accompanying drawings in which like reference numerals refer to like elements in the several figures. The following description of example embodiments does not represent all embodiments consistent with the present application.

[0067] Rather, they are only examples of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0068] The present application proposes an image generation scheme which can generate a target image based on an optimized subject image in which image quality problems have been eliminated, and can make the quality of the generated target image better, effectively improving the generation effect of a poster image. In order to facilitate understanding, some concepts related to the present application are explained as follows.

[0069] 1) AIGC (Artificial Intelligence Generated Content) refers to content generated automatically through artificial intelligence technology. AIGC can cover a variety of media forms, such as text, images, audio, video, etc. The core of AIGC is to use machine learning and deep learning technologies to enable computers to independently create new content without human direct involvement in the creation process. AIGC has a wide range of applications, from automatic writing, intelligent image generation to music creation, etc.

[0070] 2) AI (Artificial Intelligence) refers to the intelligent behavior exhibited by computer systems. AI refers to enabling computers or machines to have the ability of human intelligence, including learning, reasoning, problem solving, perception and understanding natural language. AI systems can process large amounts of data, recognize patterns and make decisions, simulating human thinking processes. The goal is to enable machines to exhibit intelligent behavior in various tasks, such as speech recognition, image recognition, natural language processing, etc.

[0071] 3) SAM (Sharpness-Aware Minimization) algorithm is a commonly used technology in the field of deep learning and natural language processing. SAM algorithm is an optimization technique aimed at improving the generalization ability of neural network models, thereby improving their performance on unseen data. SAM is mainly used to train deep learning models, which optimize model performance by considering the local sharpness of the model parameter space during training.

[0072] 4) DS-GAN (Dynamic Swapping GAN) is a variant of GAN (Generative Adversarial Network) that aims to improve the dynamic adjustment capabilities of GAN when generating high-quality samples. The main contribution of DS-GAN is to introduce a new mechanism for dynamically swapping specific parts of the generator and discriminator to improve training efficiency and the quality of generated samples.

[0073] 5) CLIP (Contrastive Language-Image Pre-training) model can receive a picture and some text description, then judge whether the text description is consistent with the content of the picture. The core of the CLIP model is that it uses a method called contrastive learning to train the model. In this method, the model learns to distinguish which is the correct pair (positive sample) and which is the randomly combined wrong pair (negative sample) from a large number of image-text pairs. CLIP can learn to recognize the relationship between specific visual concepts and corresponding text descriptions.

[0074] It should be noted that, before collecting the relevant data of the user (for example, the product image provided by the user), and during the process of collecting the relevant data of the user, the application can display a prompt interface, a pop-up window or output voice prompt information, which is used to prompt the user that the relevant data of the user is currently being collected, so that the application only starts to perform the relevant steps of obtaining the relevant data of the user after obtaining the confirmation operation of the user to the prompt interface or the pop-up window, otherwise (i.e., without obtaining the confirmation operation of the user to the prompt interface or the pop-up window), ending the relevant steps of obtaining the relevant data of the user, that is, not obtaining the relevant data of the user. In other words, all the user data collected by the application is collected under the condition that the user agrees and authorizes, and the collection, use and processing of the relevant user data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0075] FIG. 1 is a system structure diagram of an image generation method according to an embodiment of the application. As shown in FIG. 1, the system includes a requestor device 120 and a service device 130 of image generation.

[0076] The requestor device 120 can be a desktop computer, a laptop computer, a personal workstation or the like personal computer device.

[0077] The service device 130 can be a server, or a plurality of servers, or a virtualization platform, or a cloud computing service center.

[0078] The requestor device 120 and the service device 130 can be connected through a communication network. Optionally, the communication network is a wired network or a wireless network.

[0079] Optionally, the wireless network or the wired network described above uses standard communication techniques and / or protocols. The network is typically the Internet, but can also be any network, including, but not limited to, a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), a mobile, wired or wireless network, a private network, or any combination of the above. In some embodiments, techniques and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), and the like are used to represent data exchanged over the network. In addition, all or some links can be encrypted using conventional encryption technologies such as the Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPN), Internet Protocol Security (IPsec), and so on. In other embodiments, custom and / or proprietary data communication technologies can be employed in place of, or in addition to, the above techniques and technologies.

[0080] In the system shown in FIG. 1, a user can provide a first image and a background description text to the service-side device 130 through the request-side device 120, and the service-side device 130 receives the first image, performs a first segmentation operation on the first image, obtains an image corresponding to a specified object in the first image as a subject image from the first image, performs an image quality optimization operation on the subject image, and generates a target image based on the optimized subject image after the image quality optimization operation and the background description text.

[0081] It should be noted that in the embodiments of the present application, the above scheme is taken as an example of interaction between two devices, and optionally, the above scheme can also be implemented independently by a single device.

[0082] FIG. 2 is a flowchart of an image generation method according to an embodiment of the present application. The image generation method can be executed by a computer device, for example, the computer device can be the service-side device 130 shown in FIG. 1, or the computer device can also be another computer device, for example, a server, a personal computer, and the like. The image generation method can include the following steps.

[0083] Step 210: performing a first segmentation operation on the first image to obtain a subject image; the subject image is an image corresponding to a specified object in the first image.

[0084] The first image is an image containing the image corresponding to the specified object (i.e., the main image). For example, the specified object is an object of a certain type specified by the user; or an object contained within an image region defined by the user; or the object with the largest area segmented from the first image by the segmentation algorithm, etc.

[0085] Optionally, the first image can be obtained by a computer device from a database. This database can be a database on a server corresponding to the image generation method, where images are uploaded by the user in advance; or it can be a big data database, where the computer device can directly search for and obtain the first image through a search engine.

[0086] Optionally, the first image can be obtained by the user through a control terminal device that captures an image containing the subject image. The terminal device can be any electronic device with image acquisition function, such as a camera, mobile phone, or tablet computer.

[0087] The first segmentation operation is used to segment the image corresponding to the specified object (i.e., the main image) in the first image and obtain the segmented main image.

[0088] In this embodiment, the user can mark prompt points in the first image through the front-end interface. The prompt points are used to determine the main image that needs to be segmented in the first image. For example, the user can mark the prompt points in the middle area of ​​an object in the first image through the front-end interface to determine that the object is the specified object. Alternatively, the user can mark at least two prompt points in the edge area of ​​an object in the first image through the front-end interface, and take the object in the first image whose range is covered by a line segment connecting at least two prompt points as the specified object. After obtaining the specified object, the image corresponding to the specified object is obtained as the main image.

[0089] In the embodiments of this application, the first segmentation operation described above can be performed by an image segmentation model (e.g., the SAM model), which is a machine learning model that is capable of extracting the image corresponding to a specified object from an image and has image segmentation capabilities.

[0090] The image segmentation model can be trained based on the first image sample, the prompt point sample, and the labeled subject image. For example, taking the SAM model as an example, in the training process, the computer device inputs the first image sample and the prompt point sample into the SAM model, obtains a predicted subject image output by the SAM model, calculates a loss function value based on the difference between the predicted subject image and the labeled subject image, updates parameters of the SAM model based on the loss function value to complete training of the SAM model, and uses the trained SAM model as the image segmentation model.

[0091] After the user labels the prompt point in the first image, the computer device inputs the prompt point and the first image into the machine learning model. The machine learning model analyzes the first image based on the prompt point, generates a segmentation mask for a range image corresponding to the prompt point in the first image, and uses the segmentation mask to indicate a part of the first image that belongs to the subject image, and obtains the subject image.

[0092] In the embodiments of the present application, by performing the first segmentation operation, the unimportant background image (i.e., the image other than the subject image) in the first image can be removed, and the useful subject image can be retained. This avoids the problem that the subsequent computer device generates a target image for the subject image, and the quality of the generated target image is too poor due to the influence of other unimportant images. The quality of the generated target image can be effectively guaranteed.

[0093] Step 220: performing an optimization operation on the subject image to obtain an optimized subject image; the optimization operation is used to eliminate image defects introduced by the segmentation operation in the subject image.

[0094] The optimized subject image is the subject image obtained after the first image is subjected to the segmentation operation and the image defects are eliminated. The image defects are image quality problems introduced by the segmentation operation, for example, holes and discrete points in the subject image, and edge segmentation jaggies in the subject image.

[0095] In the embodiments of the present application, the optimization operation can be implemented by at least one image optimization model. The image optimization model is a machine learning model that can solve at least one of the image quality problems, such as holes and discrete points in the subject image, and edge segmentation jaggies in the subject image.

[0096] The image optimization model can be trained based on the subject image sample and the labeled optimized subject image. For example, taking the GLCIC (Globally and Locally Consistent Image Completion) model as an example, in the training process, the computer device inputs the subject image sample into the GLCIC model, obtains a predicted optimized subject image output by the GLCIC model, calculates a loss function value based on the difference between the predicted optimized subject image and the labeled optimized subject image, updates the parameters of the GLCIC model based on the loss function value to complete the training of the GLCIC model, and uses the trained GLCIC model as the image optimization model.

[0097] Optionally, the performing the optimization operation can be implemented by at least one image optimization algorithm. The image optimization algorithm is a machine learning technology that can solve at least one of the image quality problems of the holes and discrete points in the subject image and the edge segmentation sawtooth in the subject image. The image optimization algorithm can include, but is not limited to, any one of the following:

[0098] 1) bilateral filtering: bilateral filtering is used to reduce noise in an image while preserving edge details of the image, and is suitable for improving the edge segmentation sawtooth problem of the image.

[0099] 2) guided filtering: a guided image is used to help smooth the image, and is suitable for edge refinement and hole repair of the image.

[0100] 3) morphological operation: such as dilation and erosion, which can be used to fill holes or eliminate discrete points in the image.

[0101] In the embodiments of the present application, since the subject image is obtained by performing the first segmentation operation by the computer device, the first segmentation operation will cause the segmented subject image to contain image defects. By performing the optimization operation, the image quality problems in the subject image can be eliminated, the high quality of the subject image is ensured, and the quality of the target image generated by the computer device based on the optimized subject image is better.

[0102] Step 230: generating a target image based on the optimized subject image, the target image being a poster with a specified object as the subject.

[0103] In the embodiments of the present application, the target image generated by the computer device can include the optimized subject image, and can also include text or patterns for introducing the object corresponding to the optimized target subject.

[0104] In the embodiment of the present application, the computer device performs a segmentation operation on the image corresponding to the specified object in the first image to extract the subject image from the first image. Since the subject image will have image quality problems when it is segmented, the computer device performs an optimization operation on the subject image to eliminate the image quality problems introduced in the segmentation operation. The image quality of the optimized subject image obtained after the optimization operation is better, and the specified object can be displayed more clearly. When generating a target image (i.e., a poster containing the specified object) based on the optimized subject image that has eliminated the image quality problems, the specified object in the optimized subject image is clearer, so the quality of the generated target image is better, effectively improving the generation effect of the poster image.

[0105] In some embodiments, the optimization operation includes at least one of a first optimization operation and a second optimization operation: the first optimization operation is used to eliminate holes and discrete points in the subject image; and the second optimization operation is used to eliminate edge segmentation jaggies in the subject image.

[0106] The hole refers to an unfilled area or a missing pixel area inside the subject object. The hole can be caused by noise, occlusion in the image acquisition process, or defects in the first segmentation operation (e.g., a segmentation algorithm).

[0107] The discrete point refers to a single or a few pixels scattered inside or around the subject object, which has no continuity with other parts of the subject object. The discrete point is usually the result of image noise, and can also be an error generated in the process of performing the first segmentation operation on the subject image.

[0108] The edge segmentation jaggies refer to the fact that the edge of the subject image is not smooth enough and appears jagged. Since the edge of the subject image is composed of discrete pixels, the edge segmentation jaggies are easily caused when the discrete nature of the subject image is poor.

[0109] The first optimization operation can be implemented by an image optimization model that has the function of eliminating holes and discrete points in the subject image, or by an image optimization algorithm (e.g., dilation, closing operation) that can eliminate holes and discrete points in the subject image.

[0110] The second optimization operation can be implemented by an image optimization model that has the function of eliminating edge segmentation jaggies in the subject image, or by an image optimization algorithm (e.g., bilateral filtering, Gaussian blur) that can eliminate edge segmentation jaggies in the subject image.

[0111] In the embodiments of the present application, the computer device performs a first segmentation operation on the first image, and after obtaining the subject image, at least one of a first optimization operation and a second optimization operation can be performed on the subject image.

[0112] Optionally, the computer device performs the first optimization operation on the subject image to obtain an optimized subject image, which is the subject image with holes and discrete points in the image eliminated.

[0113] Optionally, the computer device performs the second optimization operation on the subject image to obtain an optimized subject image, which is the subject image with edge segmentation jaggies in the image eliminated.

[0114] Optionally, the computer device performs the first optimization operation on the subject image, and then performs the second optimization operation to obtain an optimized subject image, which is the subject image with holes, discrete points and edge segmentation jaggies in the image eliminated.

[0115] Optionally, the computer device performs the second optimization operation on the subject image, and then performs the first optimization operation to obtain an optimized subject image, which is the subject image with holes, discrete points and edge segmentation jaggies in the image eliminated.

[0116] In the embodiments of the present application, different optimization operations can be taken by the computer device for different image defects to eliminate the image defects in a targeted manner, which expands the optimization method of image defects, makes the effect of eliminating image defects better, and thus makes the optimized subject image more clearly display the specified object, which is beneficial to subsequent target image generation work and makes the effect of subsequent target image generation better.

[0117] In some embodiments, based on FIG. 2, please refer to FIG. 3, which is a flowchart of an image generation method according to an embodiment of the present application. As shown in FIG. 3, in the case where the optimization operation includes a first optimization operation, step 220 in FIG. 2 can be implemented as at least one of the following steps: step 220a, step 220b and step 220c.

[0118] Step 220a: performing a closing operation on a subject region in a segmentation mask of the subject image; the segmentation mask contains one or more subject regions and one or more non-subject regions, the subject region is a continuous region of the specified object in the first image, and the non-subject region is a continuous region in the first image other than the subject region.

[0119] The segmentation mask can be a binary image containing "0" and "1", and different values are used to represent different regions in the subject image, for example, "1" represents the subject region and "0" represents the non-subject region.

[0120] After the computer device obtains the segmentation mask of the subject image, the computer device performs an expansion operation on the subject region in the segmentation mask using the selected structural element. The expansion operation can be moving the structural element along all 1s (subject regions) in the segmentation mask, and setting all positions covered by the structural element to 1 if there is any 1 in the positions covered by the structural element. After the expansion operation, the computer device performs an erosion operation on the segmentation mask after the expansion operation using the same structural element. The erosion operation is moving the structural element along all 1s in the mask, and setting the pixel at the center position of the structural element to 1 when the structural element is completely surrounded by 1s, so that the subject region is reduced to the original size, but the holes in the subject region are filled and the boundary of the subject region is smoother.

[0121] In the embodiments of the present application, when the computer device performs a closing operation on the subject region, the computer device first performs an expansion operation on the subject region and then performs an erosion operation on the result after the expansion operation. The computer device can eliminate the narrow gaps and small holes in the image corresponding to the subject region in time, and eliminate the noise generated in the expansion operation and the noise originally existing in the image corresponding to the subject region, thereby improving the optimization effect of the subject region of the subject image.

[0122] In step 220b, for the first subject region in the segmentation mask of the subject image, if the area of the first subject region is less than the first area, the first subject region is removed. The first subject region is one of the one or more subject regions.

[0123] In the embodiments of the present application, the computer device can preset an area threshold as the first area. After the computer device obtains the first subject region in the segmentation mask of the subject image, the computer device calculates the area of the first subject region and compares the area of the first subject region with the first area. If the area of the first subject region is less than the first area, the computer device removes the first subject region, for example, sets the values of all pixels in the first subject region to 0.

[0124] Optionally, the first area can be a specified number of pixels. The computer device can obtain the number of pixels in the first subject region, each pixel representing a unit area. If the number of pixels in the first subject region is less than the specified number of pixels, the first subject region is removed.

[0125] In the embodiments of the present application, by comparing the area of the first subject region with the first area, the first subject region meeting the specified condition is removed, which can effectively remove the regions with small areas in the subject region, that is, effectively remove the discrete areas and discrete points in the subject region, and optimize the image quality of the subject image.

[0126] Step 220c: For the first non-body region in the segmentation mask of the subject image, performing a fill operation on the first non-body region in a case that the first non-body region is inside any body region; the first non-body region is one of the one or more non-body regions.

[0127] In the embodiments of the present application, the computer device takes one of the non-body regions in the segmentation mask as the first non-body region, and confirms whether it is completely surrounded by the body region. It can be ensured that each boundary point of the first non-body region is an inside point of the body region by scanning the mask image. In a case that the first non-body region is inside any body region, the computer device performs a fill operation.

[0128] In some embodiments, in a case that the area of the first non-body region is less than a second area, the computer device performs the fill operation on the first non-body region. The second area can be a preset area threshold. Alternatively, the second area can be a specified number of pixels, each pixel representing a unit area. In a case that the number of pixels in the first non-body region is less than the specified number of pixels, the computer device performs the fill operation on the first non-body region.

[0129] The fill operation described above usually adopts a Flood Fill algorithm, which can effectively fill a closed region. The specific steps are as follows:

[0130] 1) The computer device selects one pixel of the first non-body region as a starting point and marks it as a starting point to be filled. The starting pixel is added to the fill queue, indicating that the subsequent fill algorithm can start processing from this point.

[0131] 2) The computer device takes out a pixel point from the first non-body region, checks the neighborhood (up, down, left, right, and possibly diagonal neighborhood) pixels of the pixel point, and if the neighborhood pixels indicate a non-body region, marks the taken-out pixel point as a value of the body region and adds it to the fill queue for further processing.

[0132] 3) The computer device performs the same operation on each pixel point in the fill queue until the fill queue is empty. All pixels in the first non-body region are filled with the pixel value of the body region.

[0133] 4) After the filling is completed, the filling result is verified. The computer device checks the mask image to ensure that all non-body region pixels have been correctly filled and that the filled region is seamlessly connected with the body region.

[0134] In the embodiments of the present application, the segmentation operation causes the main body image to contain holes, and the filling of the non-main body region in the main body region can effectively eliminate the holes in the main body image, make the contour of the target object indicated by the main body image more complete, make the main body image after filling look more smooth and natural, and effectively improve the image generation quality.

[0135] It should be noted that the computer device performs at least one of the above steps 220a, 220b and 220c on the main body image, and obtains the processed main body image as the optimized main body image.

[0136] In the embodiments of the present application, three specific first optimization operations are expanded, and the computer device can eliminate the holes and discrete points in the main body image from different angles through different specific first optimization operations, which can effectively realize the image quality optimization operation of the main body image, make the image generated by the computer device better, and effectively improve the image generation quality of the computer device.

[0137] Based on any one or more of the above embodiments, please refer to FIG. 4, which is a flow chart of an image generation method according to an embodiment of the present application. As shown in FIG. 4, in the case where the optimization operation includes the first optimization operation, step 220 in FIG. 2 can also be implemented as the following steps: step 220d and step 220e.

[0138] Step 220d: mapping the segmentation probability value of each pixel in the first image to a first range interval; the segmentation probability value is obtained by the first segmentation operation, and the probability that the pixel belongs to the main body image.

[0139] In the embodiments of the present application, after the first image is subjected to the first segmentation operation, each pixel in the image has a segmentation probability value indicating the probability that the pixel belongs to the main body image. The greater the probability value, the greater the probability that the pixel belongs to the main body image.

[0140] The above first range interval can be a range interval used to represent the gray level, for example, 【0, 255】. 0 usually represents black (no brightness at all), 255 represents white (maximum brightness), and the intermediate values represent different degrees of gray. The computer device can multiply the segmentation probability value by 255, and use the calculated product as the mapping value. The greater the mapping value, the brighter the pixel.

[0141] In the embodiments of the present application, by mapping the segmentation probability value to the first range interval, each probability value in the segmented first image can be used as a continuous mask, and different mapping values can be displayed with different brightness, which can more clearly indicate the possible degree of each pixel belonging to the main body image, and facilitate subsequent optimization operation of the main body image.

[0142] Step 220e: mapping the segmentation probability value of the pixel in the non-edge region of the subject image to a second range interval; the non-edge region is other region in the subject image except the edge region; the second range interval is located in the first range interval.

[0143] After mapping the segmentation probability value of each pixel in the first image to the first range interval, the computer device maps the segmentation probability value of the pixel in the non-edge region of the subject image to a range interval located in the first range interval, i.e., a second interval range.

[0144] The second range interval can be a range interval used to represent a gray scale, for example, 【155, 255】. 255 represents white (maximum brightness), and the smaller the number, the more the color is biased towards black (different degrees of gray), and the lower the brightness. The computer device can map the pixel mapped to the first range interval to the second interval range through linear transformation.

[0145] In the embodiment of the present application, the computer device maps the non-edge region to a range interval with greater brightness, which can avoid the problem that part of the pixels in the subject region are not displayed due to too high transparency, and can effectively display the image of the subject region.

[0146] After mapping the segmentation probability value of the pixel in the non-edge region of the subject image to the second range interval, a new segmentation mask can be obtained, and the computer device can obtain the optimized subject image after eliminating the edge jaggies by deducting the image from the first image through the new segmentation mask.

[0147] In the embodiment of the present application, the computer device maps the probability value in the subject image to a specified range interval, so that the probability value exists in the form of continuous mapping value, and the mapping value of the non-edge region in the subject region is mapped again to a pixel value with lower transparency. Since the smaller the value mapping range, the more obvious the edge jaggies, the original value is reserved for the tiny edge region, so that the edge region can be more clearly displayed in the image. At this time, the computer device can cut out the optimized subject image, i.e., the image without edge jaggies, from the mapped subject image according to the display of the image, which expands the optimization method of image edge jaggies and ensures the quality of image generation.

[0148] Based on FIG. 2, refer to FIG. 5, which is a flow chart of an image generation method according to an embodiment of the present application. The embodiment shown in FIG. 2 further includes step 240 and step 250.

[0149] Step 240: performing a second segmentation operation on the target image to obtain a segmentation image; the segmentation image contains one or more element instances.

[0150] In the embodiments of the present application, after the computer device generates at least one target image based on the optimized subject image, the computer device can perform a secondary editing operation on the target image in response to a user operation.

[0151] The second segmentation operation is used to segment at least one element instance in the target image, and the segmented image containing the at least one element instance is obtained as a segmentation image.

[0152] In the embodiments of the present application, the second segmentation operation can be performed by an image segmentation model (for example, a SAM model), which is a machine learning model with image segmentation capability and can obtain an image corresponding to an element instance from an image.

[0153] The image segmentation model can be trained based on a target image sample and a labeled segmentation image. For example, taking the SAM model as an example, in the training process, the computer device inputs the target image sample into the SAM model, obtains a predicted segmentation image output by the SAM model, calculates a loss function value based on the difference between the predicted segmentation image and the labeled segmentation image, updates the parameters of the SAM model based on the loss function value to complete the training of the SAM model, and uses the trained SAM model as the image segmentation model.

[0154] In the embodiments of the present application, the computer device can perform the second segmentation operation on at least one element instance in the target image, so as to facilitate the user to perform a secondary editing operation on the segmentation image and dynamically adjust the effect of the generated image, thereby expanding the editable image generation mode.

[0155] Step 250: In response to the editing operation on the element instance, the element instance is edited to obtain a first edited image.

[0156] The editing operation includes but is not limited to any one of the magnification operation, the reduction operation, and the movement operation.

[0157] The computer device obtains the segmentation image containing the element instance on which the editing operation is performed as the first edited image.

[0158] In the embodiments of the present application, after the computer device performs the second segmentation operation on the target image, the segmentation image containing at least one element instance is obtained, and different editing operations can be performed on the element instance in the segmentation image, thereby expanding the secondary editing mode of the target image. The user can perform a secondary editing operation on the generated target image, so as to obtain a better final image effect and improve the efficiency of image generation.

[0159] Based on FIG. 5, refer to FIG. 6, which is a flowchart of an image generation method according to an embodiment of the present application. In the embodiment shown in FIG. 6, the method further includes steps 260, 270, and 280.

[0160] Step 260: Obtain the first feature code and the second feature code.

[0161] In the embodiments of the present application, the first feature code and the second feature code are obtained when the first edited image has a vacancy.

[0162] The first feature code is the feature code output by the control unit; the control unit is configured to perform an image quality processing operation on the first edited image, and the image quality processing operation includes retaining feature details of the first edited image in the second edited image and maintaining color consistency between the second edited image and the first edited image; the second feature code is a feature code obtained by splicing the first feature code and a down-sampling mask and adding noise; and the down-sampling mask is used to indicate a hole region in the first edited image.

[0163] For example, the down-sampling mask can be a binary image, and a pixel position with a pixel value of 1 in the binary image can indicate a position of the hole region in the first edited image, and a pixel position with a pixel value of 0 in the binary image can indicate a position of a non-hole region in the first edited image.

[0164] In the embodiments of the present application, the computer device can input the first edited image containing the vacancy region into the control unit, and obtain the feature code output by the control unit as the first feature code after the control unit performs the image quality processing operation on the first edited image.

[0165] In the embodiments of the present application, the computer device can input the first edited image containing the vacancy region into the encoder, obtain the feature code corresponding to the first edited image output by the encoder, splice the feature code corresponding to the first edited image and the down-sampling mask in a channel splicing manner, add random noise after splicing, and obtain the feature code added with the random noise as the second feature code.

[0166] Step 270: Input the first feature code and the second feature code into the diffusion model to obtain an output feature code of the diffusion model; the output feature code is a feature code obtained by decoding after the first feature code and the second feature code perform a point multiplication operation.

[0167] It should be noted that the first feature encoding and the second feature encoding are not limited to performing point operation in the diffusion model, but can also perform other operations capable of fusing the first feature encoding and the second feature encoding based on the cross attention mechanism.

[0168] Step 280: input the output feature encoding into the VAE model to obtain a second edited image of the output of the VAE model.

[0169] The VAE model is a machine learning model having the ability to convert feature encodings into images.

[0170] In the embodiments of the present application, the computer device inputs the first feature encoding capable of ensuring image quality and the second feature encoding containing noise into the diffusion model, and the diffusion model fuses the output feature encoding generated by the two feature encodings, so that the generated image corresponding to the blank area of the first edited image in the second edited image generated based on the output feature encoding can retain the feature details and colors of the first edited image, effectively improving the supplementary generation effect of the blank area in the first edited image and improving the quality of image generation by the computer device.

[0171] Based on the schemes shown in the above embodiments, in some embodiments, the computer device can determine text layout information according to the target image; the text layout information is used to indicate the type of the target text and the position of the target text in the target image; and the target text is added in the target image according to the text layout information.

[0172] In the embodiments of the present application, after obtaining the target image, the computer device can automatically generate text layout information indicating the type (such as font type, color, size, etc.) of the target text and the position of the target text in the target image, and add the target text in the target image according to the text layout information. Specifically, the computer device can generate the text layout information by using a machine learning model having a text layout capability, input the target image into the machine learning model, and the machine learning model obtains and analyzes the subject image and the background image in the target image, and outputs the text layout information based on the analysis result.

[0173] Optionally, the computer device can include a database containing at least two entries, each entry storing a type of text layout information, and each entry including an identification field for indicating the subject image corresponding to the current text layout information. After receiving the target image, the computer device identifies the subject image in the target image, queries the identification field in the database, and in the case that the identification field and the subject image in the target image match successfully, obtains the text layout information in the entry corresponding to the identification field, and adds the target text in the target image based on the obtained text layout information.

[0174] Optionally, the computer device can include a text layout template library containing at least two different text layout information. After the computer device obtains the target image, the computer device can display the target image and all text layout information in the text layout template library on the terminal device of the user. The user can perform a selection operation on a type of text layout information on the terminal device. In response to receiving the selection operation on the specified text layout information, the computer device adds target text in the target image according to the text layout information selected by the user.

[0175] In the embodiments of the present application, the computer device can automatically recognize and analyze the content of the target image, and then intelligently determine the text layout information. Not only the category of the target text can be determined, but also the appropriate position of the target text in the target image can be accurately positioned. The user does not need to manually adjust the position and style of the target text, and the perfect fusion of the target text and the target image can be achieved. The way of text layout on the target image is expanded, and the efficiency of image generation is effectively improved.

[0176] Based on FIG. 2, please refer to FIG. 7, which is a flow chart of an image generation method according to an embodiment of the present application. Step 230 in FIG. 2 can be implemented as step 230a, step 230b and step 230c.

[0177] Step 230a: input the optimized subject image into a deconstruction model to obtain the output of the deconstruction model as a reference prompt word.

[0178] The reference prompt word can be at least one description word for the optimized subject image, for example, a description word for a specified object in the optimized subject image, a description word for a background image in the optimized subject image, a description word for a specified element instance in the optimized subject image, or an extended description word based on the optimized subject image (i.e., the word meaning is related to the optimized subject image, but the description word itself does not indicate the content already contained in the optimized subject image).

[0179] In the embodiments of the present application, the deconstruction model is a machine learning model that has the ability to output text for describing an image according to the image. The deconstruction model can be trained based on the optimized subject image sample and the labeled reference prompt word. For example, taking the deconstruction model as an image description model, in the training process, the computer device inputs the optimized subject image sample into the image description model to obtain the predicted reference prompt word output by the image description model. Then, the difference between the predicted reference prompt word and the labeled reference prompt word is used to calculate the loss function value. The parameters of the image description model are updated to complete the training of the image description model, and the trained image description model is used as the deconstruction model.

[0180] Step 230b: obtaining a background description text according to the reference prompt word.

[0181] The background description text is used to describe the background image of the target image finally generated by the computer device.

[0182] In the embodiments of the present application, the computer device can automatically generate the background description text according to the reference prompt word after obtaining the reference prompt word. Specifically, the computer device can generate the background description text by a machine learning model with the ability of generating background description text. The reference prompt word is input into the machine learning model, the machine learning model obtains and analyzes the reference prompt word, and generates and outputs the background description text based on the analysis result.

[0183] Optionally, the computer device can include a database, and the database includes at least two entries, each entry storing a background description text, and each entry includes an identification field for indicating the reference prompt word corresponding to the current background description text. After receiving the reference prompt word, the computer device identifies the reference prompt word, queries the identification field in the database, and obtains the background description text in the entry corresponding to the identification field in the case of successful matching between the identification field and the reference prompt word.

[0184] Optionally, the computer device can include a background description text template library, and the library includes at least two different background description texts. After the computer device obtains the reference prompt word, the computer device can display the reference prompt word and all the background description texts in the background description text template library on the terminal device of the user. The user can perform a selection operation on a background description text on the terminal device. The computer device obtains the background description text selected by the user in response to receiving the selection operation on the specified background description text.

[0185] In the embodiments of the present application, the computer device can automatically identify and analyze the reference prompt word, and intelligently determine the background description text. The user does not need to manually edit the background description text, and the background description text can be obtained, which expands the method of obtaining the background description text and effectively improves the efficiency of image generation.

[0186] Step 230c: generating a target image according to the background description text, and the background in the target image matches the background description text.

[0187] In the embodiments of the present application, the computer device can automatically generate the target image by a machine learning model with the ability of generating corresponding images based on text. For example, the computer device inputs the optimized subject image and the background description text into the machine learning model, and obtains the target image output by the machine learning model.

[0188] Alternatively, the computer device can also generate a background image based on the background description text, and generate a target image according to the optimized subject image and the background image; for example, superimposing the optimized subject image on the background image to obtain the target image; or for example, inputting the optimized subject image and the background image into a machine learning model to obtain the target image output by the machine learning model.

[0189] In the embodiments of the present application, the computer device inputs the optimized subject image into the deconstruction model to obtain the output of the deconstruction model as the reference prompt word, converts the image information into useful features (i.e., the reference prompt word), and then generates the background description text based on the reference prompt word to guide the generation of the background image, which can make the features of the generated background image more consistent with the features of the subject image, ensure the realism of the image generation, effectively improve the efficiency of the image generation, and improve the efficiency and quality of the image processing.

[0190] In some embodiments, the above step 230b can be implemented as displaying the reference prompt word to the user; receiving the background description text input by the user.

[0191] In the embodiments of the present application, the computer device can display the reference prompt word on the terminal device of the user, and the user can create the background description text according to the displayed reference prompt word and input the created background description text into the computer device.

[0192] In the embodiments of the present application, the user can input the background description text according to the reference prompt word provided by the computer device, which expands the acquisition method of the background description text.

[0193] In some embodiments, the above step 230b can be implemented as inputting the reference prompt word into a generative language model to obtain the background description text output by the generative language model.

[0194] The above generative language model can be obtained by fine-tuning training based on reference prompt word samples and labeled background description texts, for example, taking the above generative language model as an LLM (Large Language Model) model, in the fine-tuning training process, the computer device inputs the reference prompt word sample into the LLM model to obtain the predicted background description text output by the LLM model, then calculates the loss function value based on the difference between the predicted background description text and the labeled background description text, updates the parameters of the LLM model based on the loss function value to complete the training of the LLM model, and takes the trained LLM model as the generative language model.

[0195] In the embodiments of the present application, the computer device can automatically generate the background description text through the generative language model, which expands the acquisition method of the background description text and improves the acquisition efficiency of the background description text.

[0196] Based on FIG. 2, refer to FIG. 8, which is a flowchart of an image generation method according to an embodiment of the present application. Step 230 in FIG. 2 can be implemented as steps 230d, 220e and 220f, which are as follows:

[0197] Step 230d: determining one or more candidate background templates matching the optimized subject image from the background templates.

[0198] The background templates include at least one template for generating a background image. The background templates can be downloaded from a network by the computer device or uploaded to a server corresponding to the terminal device by a user.

[0199] Optionally, the computer device can automatically calculate the matching value of the background template and the optimized subject image. The higher the matching value, the better the fusion effect of the current background template and the optimized subject image, and the better the quality of the synthesized image effect. The computer device can set a matching threshold. When the matching value reaches the matching threshold, the background template corresponding to the matching value is obtained as a candidate background template. The computer device can also obtain the background template with the highest matching value as a candidate background template.

[0200] For example, the computer device can process the background template and the optimized subject image through a machine learning model to output the matching value of the background template and the optimized subject image.

[0201] Alternatively, the computer device can calculate the matching value of the background template and the optimized subject image through a pre-set image matching algorithm. For example, the matching value of the background template and the optimized subject image can be calculated through color value and contrast.

[0202] Optionally, the computer device can pre-display the synthesized effect image of the background template and the optimized subject image on the terminal device of the user. The user can select one or more candidate background templates through the synthesized effect image of the background template and the optimized subject image.

[0203] Step 230e: randomly determining a background image from the one or more candidate background templates.

[0204] In the embodiments of the present application, the computer device can randomly determine a background image from the one or more candidate background templates through a random algorithm.

[0205] Step 230f: generating a target image according to the background image and the optimized subject image.

[0206] In the embodiment of the present application, the computer device can combine the background image and the optimized subject image through the image processing model to generate a target image. Alternatively, the optimized subject image is superimposed on the background image to obtain the target image.

[0207] In the embodiment of the present application, the computer device selects one or more candidate background templates from the background templates that match the optimized subject image, which can ensure that the background templates are consistent with the subject image in visual style. Randomly selecting a background image from the candidate background templates for subsequent synthesis can improve the automation level of image synthesis. Using the selected background image and the optimized subject image for synthesis can ensure that the subject image and the background image are integrated naturally, thereby ensuring the quality of image generation.

[0208] Based on the steps in the embodiments of FIGS. 2-8 described above, the present application provides an AI poster generation method. The embodiment can automatically generate a background image according to the main features of the goods and the text description provided by the user, such as style, scene, atmosphere, etc., and complete the customized poster with text arrangement and text rendering, and local dynamic effect.

[0209] For example, refer to FIG. 9, which shows a flowchart of an AI poster generation method according to an exemplary embodiment of the present application.

[0210] As shown in FIG. 9, the first step of the present scheme is to generate a commodity background, i.e., static canvas rendering. If the user is generally satisfied with the generated content (if not satisfied, directly regenerate), and wants to fine-tune the relative positions of the elements in the canvas, secondary editing can be performed, such as adjusting the position of a pair of chopsticks in a food poster. After the content of the poster image is completed, the poster text is filled, and the filling process includes two steps: first, text layout analysis is performed according to the image content (the text includes text, logo, and QR code, etc.), and then the corresponding text rendering is performed after obtaining the position of each text in the poster. The specific implementation steps are as follows:

[0211] Step S1: static canvas rendering.

[0212] Based on the image generation capability of the diffusion model (Stable Diffusion, SD), a complete poster is rendered according to the commodity image and the background description text. The specific steps include segmentation and extraction of the commodity main body, generation of background element prompt words, user text description polishing, and background generation.

[0213] (1) Commodity segmentation

[0214] The user can upload a product image to the server, and the image source is not limited to a photographed image, an existing poster image, etc. For a multi-item scene or an image with a complex background, the server can confirm the main body that needs to be segmented according to the prompt points labeled by the user on the front-end interface, perform segmentation using the SAM algorithm, extract the target product main body, and then the position and size of the product in the poster canvas will be randomly generated.

[0215] For example, refer to FIG. 10, which shows a schematic diagram of product main body segmentation provided by an example embodiment of the present application. Part (a) of FIG. 10 shows a product image uploaded by a user, and part (b) of FIG. 10 shows the extracted target product main body.

[0216] After the target product main body is extracted, the user can adjust it according to the needs after returning to the front end. For example, if the user adjusts the product position and size, the multiple static canvases rendered at a time will have consistent product position and size, otherwise, the product position and size in each static canvas will be randomly set to provide more choices in single user generation, and by default, 4 images are generated at a time.

[0217] Post-processing optimization of segmentation algorithm:

[0218] ①Segmentation of holes and small discrete points: may occur in images with high segmentation difficulty; closed operation (first inflation and then corrosion, to eliminate small holes in the segmentation mask) + removal of small discrete masks (area threshold) + hole filling (flood fill).

[0219] For example, refer to FIG. 11, which shows a schematic diagram of hole and small discrete point optimization for image segmentation provided by an example embodiment of the present application.

[0220] ②Segmentation of main body target edge sawtooth: directly convert the probability map output by the segmentation model to 0 / 1 truncation, which may cause sawtooth problems at the edges of smooth objects. To solve this problem, a continuous segmentation mask is used. First, map the probability value output by the model to the range [0, 255] as a continuous mask; then map the non-edge area of the target in the continuous mask to a certain range (such as [155, 255]) to avoid high transparency; and the edge (the area about 4 pixels wide at the edge of the target) retains the original continuous value (value mapping will also cause edge sawtooth problems, and the smaller the value mapping range, the more obvious the edge sawtooth, so the original value is retained for the small edge area).

[0221] For example, refer to FIG. 12, which shows a schematic diagram of sawtooth optimization provided by an example embodiment of the present application.

[0222] (2) Background element prompt word generation

[0223] The backend algorithm analyzes the product main body image extracted by segmentation to generate reference prompt words for background elements, providing reference for user description of product background environment. Technically, a BLIP (Bootstrapping Language-Image Pre-training) multi-modal model can be used to generate text description of the product main body image, and the text output by the BLIP can be segmented to extract semantic-safe concrete nouns (safety: build a sensitive word library and filter the output text). For example, input a "lemon mint tea image", the algorithm outputs "lemon, mint, tea" and other element reference prompt words.

[0224] (3) Text description polishing

[0225] Optionally, the text description (such as background elements, scenes, atmosphere, etc.) input by the user can be polished using a large language model to expand and enrich the user's prompt words, thereby improving the quality of the generated image. The optimized description still supports secondary editing.

[0226] For example, "There are some fresh vegetables in the farmland" can be polished by a large language model to "In a sunny afternoon, there are many kinds of fresh vegetables in a vast farmland, with bright leaves and full fruits. Eggplants and tomatoes hang on green plants, cucumbers hide under large leaves, and cabbage and chard are neatly arranged on the ground. There is a small river around the farmland, with clear water reflecting the blue sky and white clouds. In the distance, there is a small hill with patches of green trees. In the farmland, there are several farmers picking vegetables. They are wearing loose clothes and straw hats, holding picking tools, and working busily."

[0227] (4) Background generation

[0228] In product background generation, to prevent the edges of the product main body from generating redundant content, destroying the product form, or appearing as artifacts, a ControlNet control module based on edges is used to control the generation process, i.e. edge detection is performed on the segmentation image of the product, and the edge map is input into the control module for encoding to maintain the appearance of the product form during image generation.

[0229] For example, refer to FIG. 13, which shows a schematic diagram of edge condition control according to an example embodiment of the present application.

[0230] In the commodity background generation, the input of the diffusion model is the VAE encoded commodity image with noise. The original image is not used, which can effectively avoid the influence of unsafe or interference factors that may exist in the original background, and after the position of the commodity is changed, there is a certain probability that the original image will present obvious boundaries on the canvas. Directly using a white background / random noise background, in the case of simple prompt words and lack of detailed description, the generated background may be relatively monotonous, therefore, a background template library is designed to randomly configure a background for the commodity image with a certain probability (configuration principle: avoid background images with similar main colors and excessive contrast; background images will be randomly scaled and cropped). With reference to the ground, when the user does not perform text enrichment, the probability is 50%, and for the already enriched text, the probability is 30%.

[0231] For example, refer to FIG. 14, which shows a schematic diagram of random background generation provided by an example embodiment of the present application.

[0232] Step S2: Secondary editing.

[0233] Image generation has randomness, and in static canvas rendering, multiple images (such as 4 images) are generated by default, and the front end is returned for user selection. After the user selects a certain image (such as clicking on the image for secondary editing), the front end will send a signal to the back end, and the back end will decompose and render the canvas into movable elements and send them to the front end for display. The user can perform fine-tuning operations such as moving and scaling on each element.

[0234] In implementation, the SAM algorithm is used for segmentation to decompose all element instances in the generated canvas, so that each element has operability. As shown in FIG. 15, which shows a schematic diagram of SAM segmentation of all elements provided by an example embodiment of the present application. For the canvas vacancy caused by element movement, the diffusion model is used for supplementary generation again, and the Inpainting and Reference-only two ControlNet control units are used for control generation. These two control units can correspond to the control unit that outputs the first feature code in the above-mentioned embodiment, so as to maintain the color and content consistency of the generated part and the original image. As shown in FIG. 16, which shows a schematic diagram of secondary editing-supplementary generation provided by an example embodiment of the present application. The Inpainting control unit will try to preserve the original features and details of the image as much as possible to avoid conflicts and disharmony between the region and the surrounding environment; the Reference-only control unit takes the original image as a reference, which helps to maintain the color consistency of the generated part and the original image, and avoids color difference problems.

[0235] Step S3: Text layout analysis.

[0236] Two schemes are provided for use together:

[0237] (1) Template-based text layout

[0238] The poster text template library is pre-built by the designer, including horizontal and vertical posters such as 4:3, 16:9, 9:16, etc. The template contains text layout (including QR code, logo, etc.), corresponding font size, font, and rendering color, etc. After the user inputs the specific slogan, the text will be automatically rendered according to the template. The template is randomly matched according to the principle of avoiding shielding the goods (using the goods graph mask).

[0239] (2) Automatic layout of poster text

[0240] The advantage of the designed text layout template is that the aesthetic controllability is controllable, the font size is comprehensive, and the disadvantage is that the distribution is relatively fixed, which cannot be automatically adjusted according to the image content, and the visual integration degree is poor.

[0241] DS-GAN is an algorithm that can automatically generate text layout according to the goods graph, which adopts GAN structure and is composed of a layout generator and a discriminator. In particular, DS-GAN simulates the sequential nature of human design behavior, and integrates randomly initialized layout data into a design sequence with implicit time features.

[0242] However, DS-GAN is more targeted at e-commerce posters, and the main body of the goods occupies most of the poster frame, the background elements are simple, and the saliency detection network is used to highlight the foreground target. But for more extensive posters, in addition to the main body, there are many rendering elements with inconsistent importance in the background, so the deep detection network with more information is used instead of saliency detection, and the depth map and the original image are input into the generator.

[0243] For example, refer to FIG. 17, which shows a schematic diagram of the effect comparison of depth detection and saliency detection provided by an example embodiment of the present application.

[0244] In the embodiment of the present application, the target of network training is to learn the common design specifications from the mass of designers' creativity, but due to factors such as copyright, the collection of large-scale poster data with high quality is restricted, and therefore, the network generator used in the embodiment of the present application has the following improvements:

[0245] ① Change the visual backbone to the image encoder of CLIP, use the prior knowledge obtained by CLIP pre-training on large-scale data (use the tokens of the last layer of ViT; ImageNet pre-training is based on image classification, but CLIP can complete the mapping from image to text through contrastive learning, and detailedly describe the content of the image), and improve the image understanding ability (frozen).

[0246] ②Due to limited data, the results predicted by the trained network are not robust. The prediction results are iterated to reduce the difficulty of each step of prediction and improve the final layout effect. That is, the layout output by the network is used as input for re-prediction, and the visual feature part can reuse the first inference feature. The number of iterations is a hyperparameter TL. In order to realize this fine-tuning prediction step, the network training also uses the perturbed layout data as input (the training process does not need to iteratively calculate the layout).

[0247] Post-processing: Use simple design rules to normalize the model output, including alignment of the substrate and matching text, restriction of the logo, and two-dimensional code, etc.

[0248] For example, refer to FIG. 18, which shows a schematic diagram of the network generator structure provided by an example embodiment of the present application.

[0249] For the text elements in the layout predicted by the model, the attributes such as font and color are missing, which can be obtained by using the attribute prediction model TextPainter.

[0250] The text layout provides multiple versions for the user to choose from, which are generated from templates or models according to a certain probability. After the user selects a certain layout, the user can further edit the text position, size, color, and font in the front end.

[0251] Step S4: text rendering.

[0252] According to the template rendering

[0253] For two-dimensional codes or logo images, direct mapping is performed.

[0254] Optionally, the specific text content on the poster can be given by a language large model. The user inputs the product category, slogan style, and other prompts, and the language model will give multiple selectable slogans corresponding to the input. For example, the user inputs “pearl milk tea + simple”, which is wrapped by a preset prompt word template, and the model returns “1. Black sugar, pearl, perfect fusion; 2. Pearl milk tea, rich taste...” and the like.

[0255] In summary, the overall process of poster generation is as follows: the user uploads the product image, segments the main target, analyzes the target output background element prompt word; the user inputs the text description, performs text polishing, combines the generated size and other parameters, and sends them to the backend algorithm to complete image generation; the generated image is returned, the user selects the image, performs secondary editing, determines the image, and the backend algorithm performs text layout template matching / text layout automatic prediction. After the user inputs the poster slogan text, the text is rendered, and the poster is output.

[0256] In the embodiment of the present application, the above scheme shows a set of AI poster generation scheme, which can eliminate the image jaggies through segmentation post-processing optimization; by generating poster background element prompt words, text description reference can be provided to the user; by adding a random background, the background richness can be improved; by deconstructing the generated image for secondary editing, fine tuning in space position can be supported; at the same time, the adaptive poster text layout algorithm can be optimized; the scheme can effectively improve the AI poster generation efficiency.

[0257] The above is only one embodiment of the present application and should not be regarded as a limitation of the present application. Professionals should understand that various modifications and changes can be made to the embodiment to adapt to different application requirements. Therefore, the scope of the present application should be defined by the claims attached to the claims.

[0258] Please refer to FIG. 19, which shows a block diagram of an image generation device provided by an exemplary embodiment of the present application. The image generation device can be realized as all or part of a computer device by hardware or a combination of software and hardware, to realize all or part of the steps in the above-mentioned embodiments of FIGS. 2 to 8. As shown in FIG. 19, the data processing device includes:

[0259] A segmentation operation execution module 1901 is configured to perform a first segmentation operation on the first image to obtain a subject image; the subject image is an image corresponding to the specified object in the first image;

[0260] An optimization operation execution module 1902 is configured to perform an optimization operation on the subject image to obtain an optimized subject image; the optimization operation is used to eliminate image defects introduced by the segmentation operation in the subject image;

[0261] A target image generation module 1903 is configured to generate a target image based on the optimized subject image; the target image is a poster with the specified object as the subject.

[0262] In some embodiments, the optimization operation includes at least one of a first optimization operation and a second optimization operation:

[0263] The first optimization operation is used to eliminate holes and discrete points in the subject image;

[0264] The second optimization operation is used to eliminate edge segmentation jaggies in the subject image.

[0265] In some embodiments, when the optimization operation includes the first optimization operation, the optimization operation execution module 1902 is configured to perform a closing operation on the subject region in the segmentation mask of the subject image; the segmentation mask contains one or more subject regions and one or more non-subject regions; the subject region is a continuous region corresponding to the specified object in the first image; the non-subject region is a continuous region in the first image other than the subject region.

[0266] for the first subject region in the segmentation mask of the subject image, in a case where an area of the first subject region is less than a first area, removing the first subject region; the first subject region is one of the one or more subject regions;

[0267] for the first non-subject region in the segmentation mask of the subject image, in a case where the first non-subject region is located inside any subject region, performing a filling operation on the first non-subject region; the first non-subject region is one of the one or more non-subject regions.

[0268] In some embodiments, in a case where the optimization operation comprises a second optimization operation, the optimization operation performing module 1902 is configured to map the segmentation probability value of each pixel in the first image to a first range interval; the segmentation probability value is obtained by the first segmentation operation, and indicates a probability that the pixel belongs to the subject image;

[0269] mapping pixels in a non-edge region of the subject image to a second range interval; the non-edge region is a region other than the edge region in the subject image; the second range interval is located inside the first range interval.

[0270] In some embodiments, the apparatus further comprises:

[0271] a segmentation image obtaining module configured to perform a second segmentation operation on the target image to obtain a segmentation image; the segmentation image comprises one or more element instances;

[0272] an editing module configured to perform editing on the element instance in response to an editing operation on the element instance to obtain a first edited image;

[0273] The editing operation comprises, but is not limited to, any one of an enlargement operation, a reduction operation, and a movement operation.

[0274] In some embodiments, the apparatus further comprises:

[0275] a feature code obtaining module configured to obtain a first feature code and a second feature code;

[0276] an output feature code obtaining module configured to input the first feature code and the second feature code into the diffusion model to obtain an output feature code of the diffusion model; the output feature code is a feature code obtained by decoding after the first feature code and the second feature code perform a dot product operation;

[0277] a second edited image obtaining module configured to input the output feature code into the VAE model to obtain a second edited image output by the VAE model;

[0278] The first feature code is a feature code output by the control unit; the control unit is configured to perform an image quality processing operation on the first edited image, and the image quality processing operation includes retaining feature details of the first edited image in the second edited image and maintaining color consistency between the second edited image and the first edited image; the second feature code is a feature code obtained by splicing the feature code of the first edited image and a down-sampling mask and adding noise; and the down-sampling mask is used to indicate a hole region in the first edited image.

[0279] In some embodiments, the apparatus further includes:

[0280] a text layout information determination module configured to determine text layout information according to the target image, the text layout information being used to indicate a type of the target text and a position of the target text in the target image;

[0281] a target text adding module configured to add the target text in the target image according to the text layout information.

[0282] In some embodiments, the target image generation module 1903 is configured to input the optimized subject image into a deconstruction model to obtain an output of the deconstruction model as a reference prompt word; obtain background description text according to the reference prompt word; and generate a target image according to the background description text, the background in the target image matching the background description text.

[0283] In some embodiments, the target image generation module 1903 is configured to display the reference prompt word to a user; and receive background description text input by the user.

[0284] In some embodiments, the target image generation module 1903 is configured to input the reference prompt word into a generative language model to obtain background description text output by the generative language model.

[0285] In some embodiments, the target image generation module 1903 is configured to determine one or more candidate background templates matching the optimized subject image from background templates; randomly determine a background image from the one or more candidate background templates; and generate a target image according to the background image and the optimized subject image.

[0286] Referring to FIG. 20, FIG. 20 is a structural diagram of a computer device according to an example embodiment of the present application. The computer device 2000 includes a central processing unit (CPU) 2001, a system memory 2004 including a random access memory (RAM) 2002 and a read-only memory (ROM) 2003, and a system bus 2005 that couples the system memory 2004 to the central processing unit 2001. The computer device 2000 further includes a basic input / output system (I / O system) 2006 that helps transfer information between various devices within the computer, and a mass storage device 2007 for storing an operating system 2013, application programs 2014, and other program modules 2015.

[0287] The basic input / output system 2006 includes a display 2008 for displaying information and input devices 2009 such as a mouse, keyboard, and the like for inputting information by a user. The display 2008 and input devices 2009 are connected to the central processing unit 2001 through an input / output controller 2010 connected to the system bus 2005. The basic input / output system 2006 can also include the input / output controller 2010 for receiving and processing input from a number of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 2010 provides output to a display screen, printer, or other type of output device.

[0288] The mass storage device 2007 is connected to the central processing unit 2001 through a mass storage controller (not shown) connected to the system bus 2005. The mass storage device 2007 and its associated computer-readable media provide non-volatile storage for the computer device 2000. That is, the mass storage device 2007 can include a computer-readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0289] Without loss of generality, the computer readable medium can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, digital video disc (DVD), or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. It should be understood by those skilled in the art that computer storage media does not limit to the above-mentioned several kinds. The system memory 2004 and the mass storage device 2007 mentioned above can be collectively referred to as memory.

[0290] The computer device 2000 can be connected to the Internet or other network devices through the network interface unit 2011 connected to the system bus 2005.

[0291] The memory further includes one or more programs stored in the memory, and the central processing unit 2001 implements all or part of the steps of the methods shown in FIGS. 2 to 8 by executing the one or more programs.

[0292] In an exemplary embodiment, a chip is also provided, which includes programmable logic circuit and / or program instructions, when the chip is running on a computer device, for implementing all or part of the steps of the methods shown in the various embodiments described above.

[0293] In an exemplary embodiment, a computer program product is also provided, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor reads and executes the computer instructions from the computer readable storage medium to implement all or part of the steps of the methods shown in the various embodiments described above.

[0294] In the example embodiments, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program, which is loaded and executed by a processor to implement all or part of the steps of the method shown in the above embodiments.

[0295] A person of ordinary skill in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or by program instructing related hardware, and the program can be stored in a computer readable storage medium, and the storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0296] Those skilled in the art should realize that, in the above one or more examples, the functions described in the embodiments of the present application can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium. The computer readable medium includes a computer storage medium and a communication medium, and the communication medium includes any medium that facilitates the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0297] The above description is only optional embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image generation method characterized by, The method comprises: performing a first segmentation operation on a first image to obtain a subject image; the subject image is an image corresponding to a specified object in the first image; performing an optimization operation on the subject image to obtain an optimized subject image; the optimization operation is used to eliminate image defects introduced by the segmentation operation in the subject image; generating a target image based on the optimized subject image; the target image is a poster with the specified object as the subject.

2. The method of claim 1, wherein, The optimization operation comprises at least one of a first optimization operation and a second optimization operation: The first optimization operation is used to eliminate holes and discrete points in the subject image; The second optimization operation is used to eliminate edge segmentation jaggies in the subject image.

3. The method of claim 2, wherein, In the case where the optimization operation comprises the first optimization operation, the performing of the optimization operation on the subject image to obtain the optimized subject image comprises at least one of: performing a closing operation on a subject region in a segmentation mask of the subject image; the segmentation mask contains one or more subject regions and one or more non-subject regions, the subject region is a continuous region corresponding to the specified object in the first image, and the non-subject region is a continuous region in the first image other than the subject region; for a first subject region in the segmentation mask of the subject image, if the area of the first subject region is less than a first area, the first subject region is removed; the first subject region is one of the one or more subject regions; for a first non-subject region in the segmentation mask of the subject image, if the first non-subject region is located inside any of the subject regions, a filling operation is performed on the first non-subject region; the first non-subject region is one of the one or more non-subject regions.

4. The method of claim 2, wherein, In the case where the optimization operation comprises the second optimization operation, the performing of the optimization operation on the subject image to obtain the optimized subject image comprises: mapping a segmentation probability value of each pixel in the first image to a first range interval; the segmentation probability value is obtained by the first segmentation operation, and the pixel has a probability of belonging to the subject image; mapping the segmentation probability value of the pixel in the non-edge region of the subject image to a second range interval; the non-edge region is a region other than the edge region in the subject image; the second range interval is located inside the first range interval.

5. The method of claim 1, wherein, The method further comprises: performing a second segmentation operation on the target image to obtain a segmented image; the segmented image contains one or more element instances; in response to an editing operation on the element instance, editing the element instance to obtain a first edited image; wherein the editing operation includes but is not limited to any one of the following operations: magnification operation, reduction operation and movement operation.

6. The method of claim 5, wherein, The method further comprises: obtaining a first feature code and a second feature code; input the first feature code and the second feature code into a diffusion model to obtain an output feature code of the diffusion model; the output feature code is a feature code obtained by decoding after the first feature code and the second feature code perform a point multiplication operation; input the output feature code into a VAE model to obtain a second edited image output by the VAE model; wherein the first feature code is a feature code output by a control unit; the control unit is configured to perform an image quality processing operation on the first edited image, and the image quality processing operation includes retaining feature details of the first edited image in the second edited image and maintaining color consistency between the second edited image and the first edited image; the second feature code is a feature code obtained by splicing the first feature code of the first edited image and a down-sampling mask and then adding noise; and the down-sampling mask is used to indicate a hole region in the first edited image.

7. The method of claim 1, wherein, The method further includes: determining text layout information according to the target image; the text layout information is used to indicate a type of target text and a position of the target text in the target image; adding the target text in the target image according to the text layout information.

8. The method of claim 1, wherein, The method of generating a target image based on the optimized subject image includes: inputting the optimized subject image into a deconstruction model to obtain an output of the deconstruction model as a reference prompt word; obtaining a background description text according to the reference prompt word; generating the target image according to the background description text, wherein a background in the target image matches the background description text.

9. The method of claim 8, wherein, The method of obtaining the background description text according to the reference prompt word includes: displaying the reference prompt word to a user; receiving the background description text input by the user.

10. The method of claim 8, wherein, The method of obtaining the background description text according to the reference prompt word includes: inputting the reference prompt word into a generative language model to obtain the background description text output by the generative language model.

11. The method of claim 1, wherein, The method of generating a target image based on the optimized subject image includes: determining one or more candidate background templates matching the optimized subject image from background templates; randomly determining a background image from the one or more candidate background templates; generating the target image according to the background image and the optimized subject image.

12. An image generation apparatus characterized by comprising: The device includes: a segmentation operation execution module configured to perform a first segmentation operation on a first image to obtain a subject image; the subject image is an image corresponding to a specified object in the first image; an optimization operation execution module configured to perform an optimization operation on the subject image to obtain an optimized subject image; the optimization operation is configured to eliminate image defects introduced by the segmentation operation in the subject image; a target image generation module configured to generate a target image based on the optimized subject image; the target image is a poster with the specified object as the subject.

13. A computer device, comprising: The computer device includes a processor and a memory, and the memory stores instructions executed by the processor to implement the image generation method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The storage medium stores instructions, and the instructions are executed by a processor of the computer device to implement the image generation method in any one of claims 1 to 11.

15. A computer program product, characterised in that, The computer program product includes computer instructions stored in a computer readable storage medium; the computer instructions are read and executed by a processor of the computer device to implement the image generation method in any one of claims 1 to 11.

Images