Object set and image generation method and apparatus, electronic device, storage medium, computer program, and computer program product

Abstract

The present disclosure relates to the technical field of computers, and provided thereby are an object set and image generation method and apparatus, an electronic device, a storage medium, a computer program, and a computer program product. The object set and image generation method comprises: on the basis of a prompt information set, determining a first image set; for each candidate object in a candidate object set, on the basis of the candidate object and the prompt information set, determining a second image set that matches the candidate object, the candidate object being used for representing an image style; and on the basis of attribute information of the first image set and attribute information of the second image set corresponding to each candidate object, determining a first object set from the candidate object set.

Description

Object set and image generation method, device, electronic device, storage medium, computer program and computer program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The embodiments of the present disclosure are based on and claim the priority of Chinese patent application with application number 202310664959.4, application date June 6, 2023, and application name “Object set and image generation method, device, electronic device and storage medium”. The entire content of the Chinese patent application is hereby introduced into the present disclosure as a reference. Technical Field

[0003] The present disclosure relates to, but is not limited to, the field of computer technology, and in particular to a method, device, electronic device, storage medium, computer program, and computer program product for generating an object set and an image. Background Art

[0004] Text-to-image generation (TIG), as a key component of artificial intelligence generated content (AIGC), is gaining increasing attention and application. In implementation, users simply describe the desired content through text (i.e., prompts), and the generative model generates high-quality, semantically accurate images.

[0005] In related technologies, by adding objects (such as artist names or celebrity names) to prompts, the generative model will generate images with the style and content matching the objects, achieving improved image generation. However, these objects are usually obtained through personal judgment, community experience, and manual compilation. They are limited in number, low in accuracy, and low in efficiency. As a result, the generated results are limited, unstable, and uninterpretable.

[0006] Summary of the Invention

[0007] Embodiments of the present disclosure provide a method, apparatus, electronic device, storage medium, computer program, and computer program product for generating an object set and an image.

[0008] The technical solution of the embodiment of the present disclosure is implemented as follows:

[0009] An embodiment of the present disclosure provides a method for generating an object set, which is executed by an electronic device. The method includes:

[0010] Determine a first image set based on the prompt information set; wherein each first image in the first image set corresponds to a first prompt information in the prompt information set;

[0011] For each candidate object in the candidate object set, determining a second image set matching the candidate object based on the candidate object and the prompt information set, wherein the second image set includes at least one second image, each second image corresponds to one piece of the first prompt information, and the candidate object is used to represent the image style;

[0012] A first object set is determined from the candidate object set based on the attribute information of the first image set and the attribute information of the second image set corresponding to each candidate object.

[0013] An embodiment of the present disclosure provides an image generation method, which is executed by an electronic device. The image generation method includes:

[0014] Determine at least one target object from a second object set; wherein the second object set is obtained according to any of the above object set generation methods;

[0015] Based on the prompt information and each of the target objects, an image corresponding to the prompt information is generated.

[0016] The present disclosure provides an object set generation device, which includes:

[0017] The first determining part is configured to determine a first image set based on the prompt information set; wherein each first image in the first image set corresponds to a first prompt information in the prompt information set;

[0018] a second determining part configured to determine, for each candidate object in the candidate object set, based on the candidate object and the prompt information set, a second image set matching the candidate object, wherein the second image set includes at least one second image, each second image corresponds to one piece of the first prompt information, and the candidate object is used to represent the image style;

[0019] The first generating part is configured to determine a first object set from the candidate object set based on the attribute information of the first image set and the attribute information of the second image set corresponding to each candidate object.

[0020] The present disclosure provides an image generating device, which includes:

[0021] A third determining part is configured to determine at least one target object from a second object set; wherein the second object set is obtained according to any of the above object set generation methods;

[0022] The second generating part is configured to generate an image corresponding to the prompt information based on the prompt information and each target object.

[0023] An embodiment of the present disclosure provides an electronic device, including a processor and a memory, wherein the memory stores a computer program that can be run on the processor, and the processor implements the above method when executing the computer program.

[0024] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above method is implemented.

[0025] An embodiment of the present disclosure provides a computer program, including computer-readable codes. When the computer-readable codes are executed in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0026] An embodiment of the present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, the above method is implemented.

[0027] In an embodiment of the present disclosure, a first image set is determined based on a prompt information set; wherein each first image in the first image set corresponds to a first prompt information in the prompt information set; for each candidate object in the candidate object set, a second image set matching the candidate object is determined based on the candidate object and the prompt information set, the second image set including at least one second image, each second image corresponding to one first prompt information, and the candidate object is used to characterize the image style; based on the attribute information of the first image set and the attribute information of the second image set corresponding to each candidate object, a first object set is determined from the candidate object set. In this way, the candidate object set is automatically screened through the attribute information of the first image set and the attribute information of the second image set corresponding to each candidate object to obtain the first object set. Compared with personal sensory judgment, community experience, manual sorting, etc., it not only increases the number of objects and shortens the time to obtain objects, thereby reducing the cost of obtaining objects, but also improves the reliability and accuracy of the obtained objects. Subsequently, images are generated through the first object set, so that the images have richer and more diverse enhancement effects, reducing the limitations of using only a small number of objects that have been manually experienced to improve the generated images, and the possibility of instability and unexplainability that may be caused by blindly using a large number of objects that have not been manually verified.

[0028] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0030] FIG1 is a schematic diagram of a first implementation flow of a method for generating an object set provided by an embodiment of the present disclosure;

[0031] FIG2 is a second schematic diagram of an implementation flow of a method for generating an object set provided by an embodiment of the present disclosure;

[0032] FIG3 is a first schematic diagram of an implementation flow of an image generation method provided by an embodiment of the present disclosure;

[0033] FIG4A is a third schematic diagram of an implementation flow of a method for generating an object set provided in an embodiment of the present disclosure;

[0034] FIG4B is a schematic diagram of an aesthetic score value of an image provided by an embodiment of the present disclosure;

[0035] FIG4C is a second schematic diagram of an implementation flow of an image generation method provided by an embodiment of the present disclosure;

[0036] FIG4D is a schematic diagram of an image generated based on a prompt provided by an embodiment of the present disclosure;

[0037] FIG5 is a schematic diagram of the composition structure of an object set generation device provided by an embodiment of the present disclosure;

[0038] FIG6 is a schematic diagram of the composition structure of an image generating device provided by an embodiment of the present disclosure;

[0039] FIG7 is a schematic diagram of a hardware entity of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting the present disclosure. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0041] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0042] In the following description, the terms "first\second\third" are used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second\third" can be interchanged with the specific order or sequence where permitted, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art in the art of the present disclosure. The terms used herein are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0044] In related technologies, experienced users often add common objects when writing prompts, causing the generative model to generate similar styles and content to those objects. This improves the generation effect, but relies heavily on a small number of objects manually curated from user community experience. Repeated use of these objects in the text-to-image process, while improving the generation effect, significantly reduces the diversity of the generated results. Furthermore, while some online text-to-image websites offer a wide range of objects, users lack a clear understanding of the style and content of each object. Blindly selecting these objects can lead to suboptimal generation results, resulting in instability and unexplainable results.

[0045] The present disclosure provides an object set generation method, which automatically screens the candidate object set by using the attribute information of the first image set and the attribute information of the second image set corresponding to each candidate object to obtain a first object set. Compared with methods such as personal perceptual judgment, community experience, and manual sorting, the method not only increases the number of objects and shortens the time to obtain the objects, thereby reducing the cost of obtaining the objects, but also improves the reliability and accuracy of the obtained objects. Subsequently, images are generated by the first object set, so that the images have richer and more diverse enhancement effects, reducing the limitations of using only a small number of objects with manual experience to enhance the generated images, and the possibility of instability and unexplainability caused by blindly using a large number of objects that have not been manually verified. The method provided by the present disclosure can be executed by an electronic device, which can be various types of terminals such as laptops, tablets, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), etc., and can also be implemented as a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0046] Below, the technical solutions in the embodiments of the present disclosure will be clearly and completely described in conjunction with the drawings in the embodiments of the present disclosure.

[0047] FIG1 is a schematic diagram of a first implementation flow of a method for generating an object set provided by an embodiment of the present disclosure. As shown in FIG1 , the method includes steps S11 to S13, wherein:

[0048] Step S11: Determine a first image set based on the prompt information set; wherein each first image in the first image set corresponds to a first prompt information in the prompt information set.

[0049] Here, the prompt information set includes at least one first prompt information. The first prompt information can be any appropriate prompt information. During implementation, the first prompt information can be a text prompt information, a voice prompt information, etc. For example, the first prompt information can be a text / voice prompt information describing the attribute information of a person, a virtual object, an item, etc. Among them, the attribute information can include but is not limited to gender (for example, male, female), body type (for example, tall, short, fat, thin, etc.), appearance, etc., and the virtual object can be a model, a digital human, etc. For example, the first prompt information can be "a beautiful girl". For another example, the first prompt information can be "a student riding a bicycle".

[0050] The method for obtaining the first prompt information set may be determined according to the actual application scenario, and is not limited in this embodiment of the present disclosure.

[0051] For example, the prompt information set is input by the user through the input component of the electronic device, wherein the input component may include but is not limited to a keyboard, a mouse, a touch screen, a touch pad, an audio input device, etc.

[0052] For another example, a prompt information set sent by other devices is received.

[0053] For example, according to a preset selection rule, multiple prompt information are selected from the second prompt information set as the first prompt information set. The selection rules may include but are not limited to the default configuration of the electronic device, random, user customization, user preferences, frequency of use, application scenarios, user operation information, etc. During implementation, those skilled in the art can independently set the selection rules according to actual needs, and this disclosure does not limit them. For example, the first 100 prompt information in the second prompt information set are used as the first prompt information set. For another example, 80 prompt information are randomly selected from the second prompt information set as the first prompt information set. For another example, the first prompt information set is selected from the second prompt information set in real time according to the user's gesture. For example, different gestures / operation steps correspond to different prompt information sets. For example, when the user inputs the first gesture, the first 100 prompt information are used as the first prompt information set; when the user inputs the second gesture, 100 prompt information are randomly selected as the first prompt information set.

[0054] The second prompt information set is a prompt information set obtained by preprocessing other prompt information sets. Other prompt information sets can be obtained from multiple prompt records obtained from some associated links (for example, links to text-generated images, prompt resource links) using technologies such as crawlers. During implementation, the prompt record may include but is not limited to prompt identifiers, prompts, attribute information of generated images, random numbers, etc., that is, the prompts in each prompt record are used as prompt information in other prompt information sets. Preprocessing may include but is not limited to deduplication, length screening, named entity recognition, etc. For example, deduplication processing is performed on multiple prompts, that is, for the same N prompts, only one prompt is retained, and the remaining N-1 prompt data are deleted, where N is a positive integer greater than 1. For another example, the length of each prompt is counted, and prompts that are too long / too short are deleted. For another example, each prompt is detected using a preset named entity recognition model. If it is a preset name, the prompt is retained; otherwise, the prompt is deleted. The preset name may include but is not limited to the name of a person, the name of a virtual object, etc. In this way, by extracting the second prompt information set from a large number of prompt records, a richer and more diverse enhancement effect can be achieved for the text image.

[0055] The number of first images included in the first image set is the same as the number of first prompt information included in the first prompt information set, that is, if the first prompt information set includes M pieces of first prompt information, then the first image set includes M images, where M is a positive integer.

[0056] In some embodiments, for each first prompt in the first prompt set, a preset text-to-image model is used to generate a first image corresponding to the first prompt. The text-to-image model can be any suitable model capable of generating an image based on the prompt, such as Stable Diffusion, Guided Language to Image Diffusion for Generation and Editing (GLIDE), Midjourney, or MUSE.

[0057] Step S12: For each candidate object in the candidate object set, determine a second image set that matches the candidate object based on the candidate object and the prompt information set, wherein the second image set includes at least one second image, each second image corresponds to one of the first prompt information, and the candidate object is used to represent the image style.

[0058] Here, the candidate object set includes at least one candidate object. The candidate object can be any suitable object, such as an artist name, a celebrity name, identification information of a virtual object, etc. For example, illustrator Artgerm, Avatar, Spider-Man, etc.

[0059] The second image set includes at least each second image. The number of the second images is the same as the number of the first prompt information, that is, if the number of the first prompt information is X, then the number of the second images is X, where X is a positive integer.

[0060] In some embodiments, the step S12 of “determining, based on the candidate object and each piece of the first prompt information, at least one second image having a style matching that of the candidate object” includes step S121, wherein:

[0061] Step S121: for each first prompt information in the prompt information set, determine second prompt information based on the candidate object and the first prompt information, and determine a second image matching the candidate object based on the second prompt information.

[0062] Here, the second prompt information can be a combination of the first prompt information and the candidate object in a random order. For example, the first prompt information is "beautiful girl" and the candidate object is the illustrator "artgerm", then the second prompt information can be "beautiful girl, artgerm" or "artgerm, beautiful girl".

[0063] In some embodiments, the method for determining the second image is similar to the method for determining the first image in the aforementioned step S11. During implementation, reference may be made to the specific implementation of the aforementioned step S11.

[0064] Step S13: Determine a first object set from the candidate object set based on the attribute information of the first image set and the attribute information of the second image set corresponding to each candidate object.

[0065] Here, each candidate object in the candidate object set is screened by comparing the attribute information of the first image set with the attribute information of each second image set. The screened candidate object set is then used as the first object set. The number of candidate objects in the first object set is no greater than the number of candidate objects in the candidate object set. For example, if the candidate object set includes N candidate objects, after screening, the first object set only includes M (M no greater than N) candidate objects.

[0066] The attribute information of the image set may include, but is not limited to, a score, cross-modal similarity, etc. The score is used to evaluate the artistic expression, appeal, and beauty of the image. The cross-modal similarity is used to measure the similarity between the image and the text (i.e., the prompt information).

[0067] For example, if the first score of the first image set and the second score of the second image set meet a first preset condition, indicating that the artistic expression, appeal, and beauty of the image will decrease instead of increase after using the candidate object, then the candidate object should be deleted; otherwise, the candidate object should be included as a candidate object in the first object set. The first preset condition may include, but is not limited to, the difference between the first score and the second score being less than a preset score threshold, the second score being less than the first score, and the like.

[0068] For another example, if the second cross-modal similarity of the second image set and the first cross-modal similarity of the first image set meet a second preset condition, and if the cross-modal similarity of the images decreases instead of increases after using the candidate object, and the semantics between the generated image and the original prompt information (corresponding to the aforementioned first prompt information) become even more inconsistent, then the candidate object needs to be deleted; otherwise, the candidate object is included as a candidate object in the first object set. The second preset condition may include, but is not limited to, the difference between the first cross-modal similarity and the second cross-modal similarity being less than a preset similarity threshold, the second cross-modal similarity being less than the first cross-modal similarity, etc.

[0069] Methods for determining the score of an image set may include, but are not limited to, the score of a particular image in the set, weighting / logarithmization / exponentialization of the score of a particular image, the mean / mean square error / variance of the scores of each image in the set, or the weighted mean / mean square error / variance of the scores of each image in the set. During implementation, those skilled in the art may independently select a method for determining the score of an image set based on actual needs, and this disclosure is not limited thereto. For example, the mean of the scores of each image in the set may be used as the score of the image set.

[0070] Methods for determining the cross-modal similarity of an image set may include, but are not limited to, the cross-modal similarity of a certain image in the image set, weighting / taking the logarithm / taking the exponent of the cross-modal similarity of a certain image, the mean / mean square error / variance of the cross-modal similarity of each image in the image set, and the mean / mean square error / variance after weighting the cross-modal similarity of each image in the image set. During implementation, those skilled in the art can independently select a method for determining the cross-modal similarity of an image set based on actual needs, and the embodiments of the present disclosure are not limited thereto. For example, the mean of the cross-modal similarity of each image in the image set is used as the cross-modal similarity of the image set.

[0071] In the embodiment of the present disclosure, the candidate object set is automatically screened through the first image set and the second image set corresponding to each candidate object to obtain the first object set. Compared with personal sensory judgment, community experience, manual sorting, etc., it not only increases the number of objects and shortens the time to obtain the objects, thereby reducing the cost of obtaining the objects, but also improves the reliability and accuracy of the obtained objects. Subsequently, the image is generated through the first object set, so that the image has richer and more diverse enhancement effects, reducing the limitations of using only a small number of objects that have been manually experienced to improve the generated image, and the possibility of instability and unexplainability that may be caused by blindly using a large number of objects that have not been manually verified.

[0072] FIG2 is a second schematic diagram of an implementation flow of a method for generating an object set provided by an embodiment of the present disclosure. As shown in FIG2 , the method includes steps S21 to S24, wherein:

[0073] Step S21: Determine a first image set based on the prompt information set; wherein each first image in the first image set corresponds to a first prompt information in the prompt information set.

[0074] Step S22: For each candidate object in the candidate object set, determine a second image set that matches the candidate object based on the candidate object and the prompt information set, wherein the second image set includes at least one second image, each second image corresponds to one of the first prompt information, and the candidate object is used to represent the image style.

[0075] Here, the above steps S21 to S22 correspond to the above steps S11 to S12 respectively, and the implementation of the above steps S11 to S12 may be referred to.

[0076] Step S23: for each candidate object in the candidate object set, determine a detection result of the candidate object based on the attribute information of the first image set and the attribute information of the second image set corresponding to the candidate object.

[0077] Here, the detection result of the candidate object indicates whether the candidate object needs to be deleted from the candidate object set. In some embodiments, the detection result includes a first detection result and a second detection result, wherein the first detection result indicates that the candidate object needs to be deleted from the candidate object set, and the second detection result indicates that the candidate object does not need to be deleted from the candidate object set.

[0078] In some embodiments, the attribute information of the first image set and the attribute information of the second image set corresponding to the candidate object can be compared to obtain the detection result of the candidate object. The attribute information of the image set may include but is not limited to scores, cross-modal similarity, etc.

[0079] For example, when the first score value of the first image set and the second score value of the second image set meet the aforementioned first preset condition, the first detection result is used as the detection result of the candidate object; otherwise, the second detection result is used as the detection result of the candidate object.

[0080] For another example, when the second cross-modal similarity of the second image set and the first cross-modal similarity of the first image set meet the aforementioned second preset condition, the first detection result is used as the detection result of the candidate object; otherwise, the second detection result is used as the detection result of the candidate object.

[0081] Step S24: Determine a first object set from the candidate object set based on the detection result of each candidate object.

[0082] Here, if the detection result of each candidate object is the second detection result, the candidate object set is used as the first object set; conversely, if the detection result of at least one candidate object is the first detection result, the candidate object corresponding to each first detection result is deleted from the candidate object set, and the deleted candidate object set is used as the new candidate object set, and steps S21 to S24 are executed again in a loop until the detection result of each candidate object in the new candidate object set is the second detection result.

[0083] In some embodiments, step S24 includes at least one of step S241 and step S242, wherein:

[0084] Step S241 : When the detection result of each candidate object is the second detection result, use the candidate object set as the first object set.

[0085] Here, the detection result of each candidate object indicates that there is no need to delete the corresponding candidate object from the candidate object set, indicating that no candidate object in the candidate object set needs to be deleted in this round of optimization, and the final effective, reliable, and high-quality candidate object set is obtained.

[0086] Step S242: When the detection results of each candidate object include at least one first detection result, delete the candidate object corresponding to each first detection result from the candidate object set, and use the deleted candidate object set as a new candidate object set; determine a new first image set based on a new prompt information set; for each candidate object in the new candidate object set, determine a new second image set that matches the candidate object based on the new prompt information set and the candidate object; determine the first object set from the new candidate object set based on the attribute information of the new first image set and the attribute information of the new second image set corresponding to each candidate object.

[0087] Here, the detection results of each candidate object include at least one first detection result, indicating that some candidate objects in the candidate object set need to be deleted in this round of optimization, and the candidate object set needs to be further optimized in order to obtain a final effective, reliable, and high-quality candidate object set. In implementation, the implementation method of "determining a new first image set based on a new prompt information set; for each candidate object in the new candidate object set, determining a new second image set that matches the candidate object based on the new prompt information set and the candidate object; determining the first object set from the new candidate object set based on the attribute information of the new first image set and the attribute information of the new second image set corresponding to each candidate object" in step S242 can refer to the implementation methods of the aforementioned steps S11 to S13.

[0088] In the embodiments of the present disclosure, on the one hand, the detection result of each candidate object is determined by the attribute information of the first image set and the attribute information of the second image set corresponding to each candidate object, thereby improving the accuracy of the detection result and providing data support for the subsequent screening of candidate objects; on the other hand, the candidate object set is optimized according to each detection result, which shortens the optimization time and improves the optimization efficiency compared to methods such as personal perceptual judgment, community experience, and manual sorting, thereby reducing the optimization cost and providing reliable and accurate data support for subsequent text generation.

[0089] In some embodiments, the attribute information of the first image set includes a first scoring value, and the attribute information of the second image set includes a second scoring value; and the step S23 of “determining a detection result for the candidate object based on the attribute information of the first image set and the attribute information of the second image set corresponding to the candidate object” includes steps S231 to S233, wherein:

[0090] Step S231: Determine a first scoring value corresponding to the first image set based on the scoring value of each first image in the first image set.

[0091] Here, the score value of the first image can be obtained by any suitable scoring model, algorithm, etc. For example, the scoring model includes neural networks such as VGGNet (Visual Geometry Group Network, deep convolutional neural network), Resnet, LeNet, SSD (Single Shot MultiBox Detector, single-step multi-box target detection). For another example, the Neural Image Assessment (NIMA) algorithm is used to perform aesthetic scoring of the image. For another example, an aesthetic scoring model trained on the SAC (Simulacra Aesthetic Captions) dataset uses a CLIP (Contrastive Language-Image Pretraining, based on pre-training of contrasting text-image pairs) model to extract image features, and after inputting the image features into a multi-layer linear network layer, outputs a score value between 1 and 10 as the aesthetic scoring result of the image. During implementation, the aesthetic scoring model can give a discriminative aesthetic scoring result for images of different qualities. The higher the score value, the higher the aesthetic value of the image. During implementation, those skilled in the art can independently select the method for determining the scoring system of the first image according to actual needs, and the embodiments of the present disclosure are not limited thereto.

[0092] Methods for determining the first rating value may include, but are not limited to, the rating value of a particular image in the first image set, weighting / taking the logarithm / taking the exponential of the rating value of a particular image, the mean / mean square error / variance of the rating values ​​of each image in the first image set, or the mean / mean square error / variance of the weighted rating values ​​of each image in the first image set. During implementation, those skilled in the art may independently select a method for determining the first rating value based on actual needs, and this disclosure is not limited thereto. For example, the mean of the rating values ​​of each image in the first image set may be used as the first rating value.

[0093] Step S232: Determine a second scoring value corresponding to the second image set based on the scoring value of each second image in the second image set.

[0094] Here, the method for determining the score value of the second image is similar to the method for determining the score value of the first image in the aforementioned step S231 . During implementation, reference may be made to the implementation of the aforementioned step S231 .

[0095] Methods for determining the second rating value may include, but are not limited to, the rating value of a particular image in the second image set, weighting / taking the logarithm / taking the exponential of the rating value of a particular image, the mean / mean square error / variance of the rating values ​​of each image in the second image set, or the mean / mean square error / variance of the weighted rating values ​​of each image in the second image set. During implementation, those skilled in the art may independently select a method for determining the second rating value based on actual needs, and the present disclosure is not limited thereto. For example, the mean of the rating values ​​of each second image in the second image set may be used as the second rating value.

[0096] Step S233: Determine a detection result of the candidate object based on the first scoring value and the second scoring value.

[0097] Here, the first score value and the second score value can be compared to obtain the detection result for the candidate object. For example, if the first score value is greater than the second score value, indicating that the overall aesthetic score of the image decreases instead of increases after using the candidate object, the first detection result is used as the detection result for the candidate object; otherwise, the second detection result is used as the detection result for the candidate object.

[0098] In some embodiments, step S233 includes at least one of step S2331 and step S2332, wherein:

[0099] Step S2331: When the second score value is less than the first score value, use the first detection result as the detection result of the candidate object.

[0100] Here, the first test result indicates that the candidate object needs to be deleted from the candidate object set. For example, if the first score is 7 and the second score is 5, and the second score is less than the first score, the first test result is used as the test result for the candidate object.

[0101] Step S2332: If the second score value is not less than the first score value, use the second detection result as the detection result of the candidate object.

[0102] Here, the second detection result indicates that the candidate object does not need to be removed from the candidate object set. For example, if the first score is 7 and the second score is 8, the second score is greater than the first score, indicating that the overall aesthetic score of the image has improved after using this candidate object. In this case, the second detection result is used as the detection result for this candidate object.

[0103] In the disclosed embodiment, a first score corresponding to the first image set is determined based on the score of each first image in the first image set; a second score corresponding to the second image set is determined based on the score of each second image in the second image set; and a detection result for the candidate object is determined based on the first and second scores. By comparing the scores of different image sets, the effect of the candidate objects on the aesthetic quality of the text image can be quantitatively analyzed. Compared to relying on personal perceptual judgment, this method utilizes quantifiable, comparable, and interpretable evaluation and screening criteria, thereby improving the accuracy and reliability of the detection results.

[0104] In some embodiments, the attribute information of the first image set includes a first cross-modal similarity, and the attribute information of the second image set includes a second cross-modal similarity; and the step S23 of “determining a detection result for the candidate object based on the attribute information of the first image set and the attribute information of the second image set corresponding to the candidate object” includes steps S251 to S253, wherein:

[0105] Step S251: Determine a first cross-modal similarity corresponding to the first image set based on each first image in the first image set and first prompt information corresponding to each first image.

[0106] Here, the method for determining the first cross-modal similarity may include but is not limited to the cross-modal similarity of a certain first image in the first image set, weighting / taking the logarithm / taking the exponent of the cross-modal similarity of a certain first image, the mean / mean square error / variance of the cross-modal similarity of each first image in the first image set, the mean / mean square error / variance after weighting the cross-modal similarity of each first image in the first image set, etc. Among them, the cross-modal similarity of the first image is obtained based on the first image and the first prompt information corresponding to the first image. During implementation, those skilled in the art can independently select the method for determining the first cross-modal similarity according to actual needs, and the embodiments of the present disclosure are not limited. For example, the mean of the cross-modal similarity of each first image in the first image set is used as the first cross-modal similarity.

[0107] In some embodiments, step S251 includes steps S2511 to S2512, wherein:

[0108] Step S2511: For each first image in the first image set, determine a first similarity between the first image and first prompt information corresponding to the first image.

[0109] Here, the first similarity can be obtained using any suitable model. For example, CLIP, CNN (Convolutional Neural Network) model, RNN (Recurrent Neural Network) model, FNN (Fully Neural Work) model, etc. During implementation, those skilled in the art can independently select a method for determining the first similarity based on actual needs, and the present disclosure does not limit this.

[0110] For example, through CLIP, the first prompt information and the first image are mapped to a unified shared feature space, so that similar texts and images have similar feature representations in the feature space. During implementation, the first prompt information and the first image are both mapped to vector representations of fixed length, and then the first similarity between the two vector representations is calculated. The calculation method of the first similarity can be any suitable calculation method. For example, cosine distance, inner product, Euclidean distance, Manhattan distance, Pearson correlation coefficient, etc. During implementation, those skilled in the art can independently select the calculation method of the similarity according to actual needs, and the embodiments of the present disclosure are not limited. For example, the cosine distance is used to calculate the first similarity between the two vector representations. The higher the first similarity, the higher the degree of match between the content representing the image and the semantics of the prompt information.

[0111] Step S2512: Determine a first cross-modal similarity corresponding to the first image set based on each first similarity.

[0112] Here, the first cross-modal similarity may be determined by, but is not limited to, a certain first similarity, a weighted / logarithmic / exponential value of a certain first similarity, a mean / mean square error / variance of each first similarity, a weighted mean / mean square error / variance of each first similarity, etc. For example, the mean of each first similarity is used as the first cross-modal similarity.

[0113] Step S252: Determine a second cross-modal similarity corresponding to the second image set based on each second image in the second image set and the first prompt information corresponding to each second image.

[0114] Here, the method for determining the second cross-modal similarity is similar to the method for determining the first cross-modal similarity in the aforementioned step S251 . During implementation, reference may be made to the implementation of the aforementioned step S251 .

[0115] In some embodiments, step S252 includes steps S2521 to S2522, wherein:

[0116] Step S2521: For each second image in the second image set, determine a second similarity between the second image and the first prompt information corresponding to the second image.

[0117] Here, the method for determining the second similarity is similar to the method for determining the first similarity in the aforementioned step S2511 . During implementation, reference may be made to the implementation of the aforementioned step S2511 .

[0118] Step S2522: Determine a second cross-modal similarity corresponding to the second image set based on each second similarity.

[0119] Here, the method for determining the second cross-modal similarity is similar to the method for determining the first cross-modal similarity in the aforementioned step S2512 . During implementation, reference may be made to the implementation of the aforementioned step S2512 .

[0120] Step S253: Determine a detection result of the candidate object based on the first cross-modal similarity and the second cross-modal similarity.

[0121] Here, the first cross-modal similarity and the second cross-modal similarity can be compared to obtain a detection result for the candidate object. For example, if the first cross-modal similarity and the second cross-modal similarity meet the second preset condition, the first detection result is used as the detection result for the candidate object; otherwise, the second detection result is used as the detection result for the candidate object.

[0122] In some embodiments, step S253 includes steps S2531 to S2533, wherein:

[0123] Step S2531: Determine a third cross-modal similarity based on the first cross-modal similarity and the second cross-modal similarity.

[0124] Here, the method for determining the third cross-modal similarity may include but is not limited to the first difference between the first cross-modal similarity and the second cross-modal similarity, the weighting / exponentiation / logarithm of the first difference, the second difference after weighting the first cross-modal similarity and the second cross-modal similarity, the weighting / exponentiation / logarithm of the second difference, etc. During implementation, those skilled in the art can independently select the method for determining the third cross-modal similarity according to actual needs, and the embodiments of the present disclosure are not limited thereto. For example, the first difference between the first cross-modal similarity and the second cross-modal similarity is used as the third cross-modal similarity. For example, if the first cross-modal similarity is C and the second cross-modal similarity is C_k, then the third cross-modal similarity is (C_k-C).

[0125] Step S2532: When the third cross-modal similarity is less than a similarity threshold, use the first detection result as the detection result of the candidate object.

[0126] Here, the similarity threshold may be pre-set. In implementation, the similarity threshold may be an empirical value, a value obtained through multiple experiments, or the like. In implementation, if (C_k-C) is less than δ (corresponding to the aforementioned similarity threshold), then this indicates that after using the candidate object, the cross-modal similarity of the image decreases instead of increases, and the semantics of the generated second image and the original prompt information become increasingly inconsistent. Therefore, the candidate object needs to be deleted, and the first detection result is used as the detection result for the candidate object.

[0127] Step S2533: When the third cross-modal similarity is not less than the similarity threshold, use the second detection result as the detection result of the candidate object.

[0128] Here, if (C_k-C) is not less than δ, it means that after using the candidate object, the cross-modal similarity of the image has increased, and the semantics between the generated second image and the original prompt information are closer. The candidate object needs to be retained, so the second detection result is used as the detection result of the candidate object.

[0129] In an embodiment of the present disclosure, a first cross-modal similarity corresponding to the first image set is determined based on each first image in the first image set and the first prompt information corresponding to each first image; a second cross-modal similarity corresponding to the second image set is determined based on each second image in the second image set and the first prompt information corresponding to each second image; and a detection result for the candidate object is determined based on the first cross-modal similarity and the second cross-modal similarity. In this way, by comparing the cross-modal similarities of different image sets, the semantic influence of the candidate object on the text image is quantitatively analyzed. Compared with relying on personal perceptual judgment, the accuracy and reliability of the detection results are improved by using quantifiable, comparable, and interpretable evaluation and screening criteria.

[0130] FIG3 is a schematic diagram of a first implementation flow of an image generation method provided by an embodiment of the present disclosure. As shown in FIG3 , the method includes steps S31 to S32, wherein:

[0131] Step S31: Determine at least one target object from the second object set.

[0132] Here, the second object set is obtained according to any of the above-mentioned object set generation methods. The number of target objects can be at least one. During implementation, the number of target objects can be a random number, for example, 1, 2, 3, etc.

[0133] The method of determining the target object may include, but is not limited to, random, customized, user preferences, frequency of use, user operation information, etc. During implementation, those skilled in the art may independently select the method of determining the target object according to actual needs, and this disclosure does not limit it. For example, a random number (for example, 2, 3, etc.) of target objects are randomly selected from the second object set. For another example, each object in the second image set is sorted according to the frequency of use, and the top three objects in terms of frequency of use are respectively used as a target object. For another example, the target object is determined in real time based on the user gesture. For example, different gestures correspond to different target objects, that is: when the user inputs the first gesture, the first object in the second object set is used as the target object. During implementation, each object in the second object set can be sorted according to name, size, time (for example, modification time, creation time, etc.), frequency of use, etc.; when the user inputs the second gesture, the last two objects in the second object set are respectively used as a target object. For another example, different operation steps correspond to different target objects. That is, when the operation step falls within a first length range, the first three objects in the second object set are each used as a target object; when the operation step falls within a second length range, the last two objects in the second object set are each used as a target object, where the first length range and the second length range are different. During implementation, those skilled in the art can independently set the correspondence between operation gestures, target objects, and the number of target objects based on actual needs, and this disclosure does not limit this.

[0134] Step S32: Based on the prompt information and each target object, generate an image corresponding to the prompt information.

[0135] Here, the prompt information can be any appropriate prompt information. In implementation, the prompt information can be text prompt information, voice prompt information, etc. For example, the prompt information can be text / voice prompt information describing the attributes of a person, virtual object, item, etc. For example, the prompt information can be "a tall and thin boy." For another example, the prompt information can be "a girl wearing a necklace." The prompt information can be obtained in various ways, including but not limited to inputting it through an input component, receiving it from other devices, etc.

[0136] During implementation, the prompt information and each target object are randomly combined to obtain a third prompt information. Using the aforementioned text-to-image model, a third image corresponding to the third prompt information is generated, and the third image is used as the image corresponding to the prompt information. For example, if the prompt information is "beautiful girl," and the target objects may include illustrator "artgerm," Polish artist "greg rutkowski," and artist "alphonse mucha," then the third prompt information may be "beautiful girl, artgerm, greg rutkowski, greg rutkowski," "beautiful girl, greg rutkowski, artgerm, greg rutkowski," "beautiful girl, greg rutkowski, greg rutkowski, artgerm," etc. During implementation, different third prompt information generates different enhancement effects for the generated images.

[0137] In the disclosed embodiment, at least one target object is determined from the second object set; based on the prompt information and each target object, an image corresponding to the prompt information is generated. In this way, on the one hand, since the second object set contains a rich variety of objects, compared with using only a few objects to generate images, using the second image set to generate images allows the images to have richer and more diverse enhancement effects, reducing the limitations of using only a small number of objects that have been manually experienced to enhance the generated images, and reducing the possibility of instability and unexplainability that may be caused by blindly using a large number of objects that have not been manually verified; on the other hand, a random number of target objects are automatically randomly selected from the second object set, and the user does not need to explicitly select the target object to obtain the image after artistic enhancement. The entire generation process is completely imperceptible to the user, simplifying the operation steps while also improving the user's operating experience.

[0138] The following describes the application of the image generation method provided by the embodiment of the present disclosure in actual scenarios, taking the scenario of generating an image based on prompt information and the artist name (corresponding to the aforementioned target object) as an example.

[0139] As an important component of AIGC, text-generated images have received increasing attention and application with the continuous improvement and breakthroughs in their generation quality. Users only need to describe the expected content through text (i.e., prompt information), and the text-generated image model can generate high-quality image content that meets the semantic requirements of the prompt information. A common technique to improve the generation effect is to use the artist's name in the prompt information. Since the text-generated image model is trained based on a large amount of training data containing image and text pairs, and these training data contain works of various artists, the trained text-generated image model can learn the mapping relationship between the artist's name and his or her works. When the prompt information contains a certain artist's name, the text-generated image model will accordingly generate a style and content similar to the artist's work, thereby achieving the purpose of improving the generation effect.

[0140] In related technologies, experienced text-to-image users often add common artist names when writing prompts (i.e., prompt information). This allows the text generation model to generate images with a style and content similar to that artist, thus improving the generation effect. However, these artist names rely primarily on a very limited number of artists obtained through user community experience and manual curation. Repeated use of these artists in the text-to-image process improves the image generation effect but seriously affects the diversity of the generated image results. Meanwhile, some online text-to-image websites offer a wide range of artist names for users to use, but users do not understand the style and content of each artist. Blindly selecting these names can lead to suboptimal generation results, resulting in instability and unexplainability.

[0141] The disclosed embodiments provide an image generation method, which extracts a large number of candidate artist names (corresponding to the aforementioned candidate object set) from a large amount of prompt data, and uses a pre-trained model with the help of quantitative evaluation indicators (corresponding to the aforementioned score value and cross-modal similarity) to screen out a high-quality artist name set (corresponding to the aforementioned first object set) that can effectively improve the generation effect without affecting the semantic requirements of the original prompt. The artist name set is used to improve the image generation effect, thereby resolving the limitations of using only a small number of manually verified artist names to improve the generation effect, and the instability and unexplainability that may be caused by blindly using a large number of unverified artist names.

[0142] The following describes the implementation process of the image generation method provided by the embodiment of the present disclosure from the two stages of the screening stage and the use stage, wherein:

[0143] 1. The screening stage mainly screens the candidate artist name set (corresponding to the aforementioned candidate object set) to obtain the target artist name set (corresponding to the aforementioned first image set). FIG4A is a third schematic diagram of the implementation flow of an object set generation method provided by an embodiment of the present disclosure. As shown in FIG4A , the method includes steps S401 to S407, wherein:

[0144] Step S401: Determine a set of candidate artist names based on a large number of prompts;

[0145] Unlike the prior art methods that primarily rely on manual collation, in order to obtain as many and comprehensive candidate artist names as possible, the present invention obtains a large number of prompt records from multiple widely used online text-to-image websites and prompt resource integration websites. Each prompt record includes identification information, prompt words, image attribute information, random number seeds, etc. The prompt words in each prompt record are obtained from the large number of prompt records to form a first prompt set. To improve the quality of the prompt words, overly long and short prompt words are deleted from the first prompt set to form a second prompt set. A named entity recognition model is used to identify each prompt word in the second prompt set, and prompt words that do not include names are deleted from the second prompt set to form a candidate artist name set. If the artist name set is manually processed one by one, for example, by searching each artist's artwork through a search engine and judging whether to retain the artist based on personal intuition or testing its effectiveness on a text-to-image model, it would consume a lot of time and manpower. In contrast, a fully automated process, relying on a pre-trained model and quantitative evaluation indicators to complete the screening process, is more efficient and produces more reliable results.

[0146] Step S402: determining an evaluation set (corresponding to the aforementioned prompt information set) from a large number of prompts;

[0147] Here, since the second prompt set contains a large number of prompts, it is not feasible to filter the artist name set based on the complete second prompt set. Therefore, multiple rounds of screening can be performed, and in each round of screening, a small number of prompts (for example, 100) are randomly selected from the second prompt set as the evaluation set.

[0148] Step S403: using the text-to-image model, based on each prompt in the evaluation set, determine a benchmark set (corresponding to the aforementioned first image set);

[0149] Here, for each prompt in the evaluation set, a text-to-image model (e.g., Stable Diffusion) is used to generate an image corresponding to each prompt (corresponding to the aforementioned first image), and the resulting image set is used as the benchmark set.

[0150] Step S404: For each candidate artist name in the candidate artist name set, the candidate artist name is concatenated with each original prompt in the evaluation set to obtain a corresponding new prompt (corresponding to the aforementioned second prompt information). Using the text-to-image model, an image corresponding to each new prompt is generated (corresponding to the aforementioned second image). The image set formed by the images corresponding to each new prompt serves as the comparison set corresponding to the candidate artist name (corresponding to the aforementioned second image set).

[0151] To verify and evaluate the impact of each artist name in the artist name set on the text-generated image, we concatenate each artist name to each original prompt in the evaluation set before performing text-generated image processing. The resulting image set serves as the comparison set. For example, assuming the artist name set includes K candidate names, K comparison sets will be generated, each containing an equal number of images. In practice, by calculating the results of various evaluation metrics for the baseline set and each comparison set, we can quantitatively analyze the impact of each artist name on the text-generated image.

[0152] Step S405: using a preset aesthetic scoring model, determining the aesthetic score of the reference set (corresponding to the aforementioned first scoring value) and the aesthetic score of each comparison set (corresponding to the aforementioned second scoring value);

[0153] Here, generally speaking, the result of text-to-image generation should be as beautiful as possible, with strong artistic expression and appeal, which can bring aesthetic pleasure to the viewer and make them willing to share it with others. However, the judgment of beauty is a very subjective task, and different people often have different understandings and judgment standards of beauty. However, under the premise of sufficient data volume, it is still possible to rely on the model to learn a relatively reliable, stable and accurate aesthetic scoring ability. In this disclosure, an aesthetic scoring model trained on the SAC dataset is used to calculate the aesthetic score of each image in the benchmark set and the aesthetic score of each image in each comparison set, and the average of the aesthetic scores of each image in the benchmark set is used as the aesthetic score A of the benchmark set, and the average of the aesthetic scores of each image in the comparison set is used as the aesthetic score A_k of the comparison set, so as to judge whether the use of a certain artist's name can effectively improve the aesthetic quality of the image.

[0154] FIG4B is a schematic diagram of an aesthetic scoring value for an image provided by an embodiment of the present disclosure. As shown in FIG4B , each image 41 is scored using an aesthetic scoring model to obtain a score 42 corresponding to each image 41. In implementation, differentiating scores 42 can be assigned to images 41 of varying quality. A higher score 42 indicates a higher aesthetic value for the image 41.

[0155] Step S406: using the large-scale cross-modal pre-training model, respectively determining the cross-modal similarity of the benchmark set (corresponding to the aforementioned first cross-modal similarity) and the cross-modal similarity of each comparison set (corresponding to the aforementioned second cross-modal similarity);

[0156] Here, using the artist's name in the prompt can improve the quality, but sometimes it will have a serious impact on the original semantics. For example, the original prompt is "a boy", but the content of a certain artist's works is mainly landscape. Then, after splicing the artist's name after the original prompt, the generated result may not contain the boy at all, but instead generate a landscape picture. This situation changes the original semantics and will seriously affect the user's text-to-image experience. During implementation, the artist's name cannot cause changes or impacts to the original semantics. In order to calculate the cross-modal similarity between the generated image and the original prompt, the present disclosure uses CLIP to map both the original prompt and the generated image into a fixed-length vector representation, and then calculates the cosine similarity between the two. The higher the similarity, the more the content of the generated image meets the semantic requirements of the original prompt. During implementation, CLIP is used to calculate the cross-modal similarity of each image in the baseline set and the cross-modal similarity of each image in each comparison set, and the average cross-modal similarity of each image in the baseline set is used as the cross-modal similarity C of the baseline set, and the average cross-modal similarity of each image in the comparison set is used as the cross-modal similarity C_k of the comparison set, so as to determine whether the use of a certain artist's name will affect the semantics of the original prompt.

[0157] Step S407 : Based on the aesthetic score of the baseline set and the aesthetic score of each comparison set, and the cross-modal similarity of the baseline set and the cross-modal similarity of each comparison set, the candidate artist name set is screened to obtain a target artist name set (corresponding to the aforementioned first object set).

[0158] Here, we determine whether the aesthetic score A_i of the i-th comparison set is greater than the aesthetic score A of the benchmark set. If A_i is less than A, it indicates that the overall aesthetic score of the generated image decreases instead of increases after using the i-th artist name. In this case, we remove the artist name from the optimization round. Otherwise, we retain the artist name. Here, i is not greater than K, where K is the total number of artist names in the artist name set.

[0159] Determine whether the cross-modal similarity C_i of the i-th comparison set is greater than the cross-modal similarity C of the benchmark set. If C_i minus C is less than the set threshold, it indicates that after using the i-th artist name, the overall cross-modal similarity of the generated image has decreased, and the generated content is more inconsistent with the semantics of the original prompt. In this case, the artist name needs to be deleted in this round of optimization. Otherwise, the artist name needs to be retained in this round of optimization.

[0160] In some implementations, the aesthetic score of the comparison set and the aesthetic score of the benchmark set may be determined first. If the aesthetic score of the comparison set is not less than the aesthetic score of the benchmark set, the cross-modal similarity of the comparison set and the cross-modal similarity of the benchmark set may be determined.

[0161] In some embodiments, the cross-modal similarity of the comparison set and the cross-modal similarity of the benchmark set can be determined first. If the cross-modal similarity of the comparison set minus the cross-modal similarity of the benchmark set is less than a set threshold, the aesthetic score of the comparison set and the aesthetic score of the benchmark set can be determined.

[0162] If no artist names are deleted in this round of optimization, the optimization work is stopped; otherwise, the above steps S402 to S407 are executed to carry out the next round of optimization until no artist names are deleted in a certain round of optimization, and the final artist name set is used as a valid, reliable, and high-quality target artist name set (corresponding to the aforementioned first object set).

[0163] 2. The use phase mainly uses the target artist name set to generate an image. FIG4C is a second schematic diagram of the implementation process of an image generation method provided by an embodiment of the present disclosure. As shown in FIG4C , the method includes steps S411 to S412, wherein:

[0164] Step S411: determining at least one target artist name (corresponding to the aforementioned target object) from the target artist name set (corresponding to the aforementioned second object set);

[0165] Here, in some embodiments, several (eg, 1 to 3) artist names are randomly selected from the target artist name set as each target artist name.

[0166] Step S412: Based on the prompt and the name of each target artist, generate an image corresponding to the prompt.

[0167] Here, each target artist's name is appended to the user's prompt, and a text-to-image model is used to generate an image corresponding to that prompt. The entire generation process is completely invisible to the user; they don't need to explicitly select an artist name to obtain an artistically enhanced image. Because the artist name set includes multiple names and the number of randomly selected names is not fixed, the generated images are more diverse and richer than using only a small number of artist names, resulting in repetitive content and style. This significantly improves both the quality of details and the overall aesthetic.

[0168] FIG4D is a schematic diagram of an image generated based on a prompt provided by an embodiment of the present disclosure. As shown in FIG4D , after the user inputs the prompt 431:

[0169] If the target artist name is empty, that is, art enhancement is not used, the text-to-image model 46 is used to generate an image 432 corresponding to the prompt 431;

[0170] If the target artist names include artist name 1 (John Fabian Carlson) and artist name 2 (Charles Harold Davis), each target artist name is concatenated with prompt 431 to form a new prompt 441. Then, using text-to-image model 46, an image 442 corresponding to prompt 441 is generated and used as the image corresponding to prompt 431. Image 442 has a style and content similar to those of the two artists, "John Fabian Carlson" and "Charles Harold Davis."

[0171] If the target artist names include artist name three (Ilya Kuvshinov), artist name four (Frans Koppelaar), and artist name five (Harriet Backer), after each target artist name is spliced ​​into prompt 431, a new prompt 451 is formed. Then, using text-to-image model 46, an image 452 corresponding to prompt 451 can be generated, and image 452 can be used as the image corresponding to prompt 431, wherein image 452 has a style and content similar to those of the three artists "Ilya Kuvshinov", "Frans Koppelaar", and "Harriet Backer".

[0172] The method provided by the embodiment of the present disclosure has at least the following beneficial effects:

[0173] 1) Extracting a large number of artist names from a massive amount of prompt records. Compared to relying primarily on manually curating a small number of available artist names, which can lead to repetitive and similar generated content, the optimized artist name set contains a large number of artist names, and the number of randomly selected artist names is not fixed, achieving a richer and more diverse art enhancement effect.

[0174] 2) Using aesthetic scoring models and CLIP models trained on a large amount of annotated data to calculate evaluation indicators for different image sets, compared with relying mainly on personal perceptual judgment for screening, which results in a limited number of screening results and is subjective and unreliable, the evaluation criteria and screening criteria are quantifiable, comparable, and explainable, resulting in a larger number of screening results that are more objective and reliable.

[0175] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0176] Based on the above embodiments, an embodiment of the present disclosure provides an object set generation device. FIG5 is a schematic diagram of the composition structure of an object set generation device provided by an embodiment of the present disclosure. As shown in FIG5 , the object set generation device 50 includes a first determination part 51, a second determination part 52, and a first generation part 53, wherein:

[0177] The first determining part 51 is configured to determine a first image set based on the prompt information set; wherein each first image in the first image set corresponds to a first prompt information in the prompt information set;

[0178] The second determining portion 52 is configured to determine, for each candidate object in the candidate object set, a second image set matching the candidate object based on the candidate object and the prompt information set, wherein the second image set includes at least one second image, each second image corresponds to one piece of the first prompt information, and the candidate object is used to represent the image style;

[0179] The first generating part 53 is configured to determine a first object set from the candidate object set based on the attribute information of the first image set and the attribute information of the second image set corresponding to each candidate object.

[0180] In some embodiments, the first generating part 53 is further configured to: for each candidate object in the candidate object set, determine the detection result of the candidate object based on the attribute information of the first image set and the attribute information of the second image set corresponding to the candidate object; wherein the detection result of each candidate object represents whether the corresponding candidate object needs to be deleted from the candidate object set; based on the detection result of each candidate object, determine the first object set from the candidate object set.

[0181] In some embodiments, the attribute information of the first image set includes a first scoring value, and the attribute information of the second image set includes a second scoring value; the first generating part 53 is further configured to: determine the first scoring value corresponding to the first image set based on the scoring value of each first image in the first image set; determine the second scoring value corresponding to the second image set based on the scoring value of each second image in the second image set; and determine the detection result of the candidate object based on the first scoring value and the second scoring value.

[0182] In some embodiments, the first generating part 53 is further configured to do at least one of the following: when the second score value is less than the first score value, use the first detection result as the detection result of the candidate object; wherein the first detection result indicates that the candidate object needs to be deleted from the candidate object set; when the second score value is not less than the first score value, use the second detection result as the detection result of the candidate object; wherein the second detection result indicates that the candidate object does not need to be deleted from the candidate object set.

[0183] In some embodiments, the attribute information of the first image set includes a first cross-modal similarity, and the attribute information of the second image set includes a second cross-modal similarity; the first generating part 53 is further configured to: determine the first cross-modal similarity corresponding to the first image set based on each first image in the first image set and the first prompt information corresponding to each first image; determine the second cross-modal similarity corresponding to the second image set based on each second image in the second image set and the first prompt information corresponding to each second image; determine the detection result of the candidate object based on the first cross-modal similarity and the second cross-modal similarity.

[0184] In some embodiments, the first generating portion 53 is further configured to: determine, for each first image in the first image set, a first similarity between the first image and the first prompt information corresponding to the first image; and determine, based on each first similarity, a first cross-modal similarity corresponding to the first image set;

[0185] In some embodiments, the first generating part 53 is further configured to: determine, for each second image in the second image set, a second similarity between the second image and the first prompt information corresponding to the second image; and determine a second cross-modal similarity corresponding to the second image set based on each second similarity.

[0186] In some embodiments, the first generating part 53 is further configured to: determine a third cross-modal similarity based on the first cross-modal similarity and the second cross-modal similarity; when the third cross-modal similarity is less than a similarity threshold, use the first detection result as the detection result of the candidate object; when the third cross-modal similarity is not less than the similarity threshold, use the second detection result as the detection result of the candidate object.

[0187] In some embodiments, the first generating part 53 is further configured to be at least one of the following: when the detection result of each candidate object is the second detection result, the candidate object set is used as the first object set; when the detection result of each candidate object includes at least one first detection result, the candidate object corresponding to each first detection result is deleted from the candidate object set, and the deleted candidate object set is used as a new candidate object set; based on the new prompt information set, a new first image set is determined; for each candidate object in the new candidate object set, based on the new prompt information set and the candidate object, a new second image set that matches the candidate object is determined; based on the attribute information of the new first image set and the attribute information of the new second image set corresponding to each candidate object, the first object set is determined from the new candidate object set.

[0188] In some embodiments, the second determining portion 52 is further configured to: for each first prompt information in the prompt information set, determine second prompt information based on the candidate object and the first prompt information, and determine a second image matching the candidate object based on the second prompt information.

[0189] The description of the above embodiment of the object set generation apparatus is similar to the description of the above embodiment of the object set generation method, and has similar beneficial effects as the embodiment of the object set generation method. For technical details not disclosed in the embodiment of the object set generation apparatus of this disclosure, please refer to the description of the embodiment of the object set generation method of this disclosure for understanding.

[0190] In the embodiments of the present disclosure and other embodiments, "part" can be part of a circuit, part of a processor, part of a program or software, etc., and of course it can also be a unit, a module, or a non-modular one.

[0191] Based on the above embodiments, an image generating device is provided in an embodiment of the present disclosure. FIG6 is a schematic diagram of the composition structure of an image generating device provided in an embodiment of the present disclosure. As shown in FIG6 , the image generating device 60 includes a third determining part 61 and a second generating part 62, wherein:

[0192] The third determining part 61 is configured to determine at least one target object from a second object set; wherein the second object set is obtained according to any of the above object set generation methods;

[0193] The second generating part 62 is configured to generate an image corresponding to the prompt information based on the prompt information and each target object.

[0194] The description of the above embodiment of the image generation device is similar to the description of the above embodiment of the image generation method, and has similar beneficial effects as the embodiment of the image generation method. For any technical details not disclosed in the embodiment of the image generation device of this disclosure, please refer to the description of the embodiment of the image generation method of this disclosure for understanding.

[0195] It should be noted that, in the embodiments of the present disclosure, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present disclosure are not limited to any specific combination of hardware and software.

[0196] An embodiment of the present disclosure provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the above method when executing the computer program.

[0197] The present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements the above method when executed by a processor. The computer-readable storage medium may be transient or non-transient.

[0198] The present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, implements some or all of the steps in the above method. In some embodiments, the computer program product can be implemented by hardware, software, or a combination thereof. In an optional embodiment, the computer program product is embodied as, for example, a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0199] It should be noted that FIG7 is a schematic diagram of a hardware entity of an electronic device in an embodiment of the present disclosure. As shown in FIG7 , the hardware entity of the electronic device 700 includes: a processor 701, a communication interface 702, and a memory 703, wherein:

[0200] The processor 701 generally controls the overall operations of the electronic device 700 .

[0201] The communication interface 702 enables the electronic device to communicate with other terminals or servers through a network.

[0202] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or processed by the processor 701 and various parts of the electronic device 700 (for example, image data, audio data, voice communication data, and video communication data). This can be implemented using flash memory (FLASH) or random access memory (RAM). Data can be transmitted between the processor 701, the communication interface 702, and the memory 703 via the bus 704.

[0203] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present disclosure, please refer to the description of the method embodiments of the present disclosure for understanding.

[0204] It should be understood that “one embodiment” or “an embodiment” mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, “in one embodiment” or “in an embodiment” appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The serial numbers of the embodiments of the present disclosure mentioned above are for description and do not represent the advantages and disadvantages of the embodiments.

[0205] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0206] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are schematic. For example, the division of the units is a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0207] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0208] In addition, all functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0209] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0210] Alternatively, if the above-mentioned integrated unit of the present disclosure is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0211] The above is only an embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present disclosure, and they should all be covered by the protection scope of the present disclosure. Industrial Applicability

[0212] The embodiments of the present disclosure disclose an object set and image generation method, apparatus, electronic device, storage medium, computer program and computer program product, wherein the object set and image generation method includes: determining a first image set based on a prompt information set; wherein each first image in the first image set corresponds to a first prompt information in the prompt information set; for each candidate object in the candidate object set, determining a second image set matching the candidate object based on the candidate object and the prompt information set, the second image set including at least one second image, each second image corresponding to a first prompt information, and the candidate object being used to characterize the image style; determining a first object set from the candidate object set based on the attribute information of the first image set and the attribute information of the second image set corresponding to each candidate object. In the embodiment of the present disclosure, the candidate object set is automatically screened through the attribute information of the first image set and the attribute information of the second image set corresponding to each candidate object to obtain the first object set. Compared with personal sensory judgment, community experience, manual sorting, etc., it not only increases the number of objects and shortens the time to obtain objects, thereby reducing the cost of obtaining objects, but also improves the reliability and accuracy of the obtained objects. Subsequently, images are generated through the first object set, so that the images have richer and more diverse enhancement effects, reducing the limitations of using only a small number of objects that have been manually experienced to improve the generated images, and the possibility of instability and unexplainability that may be caused by blindly using a large number of objects that have not been manually verified.

Claims

1. A method for generating an object set, the method comprising: Based on the prompt information set, determining a first image set; wherein each first image in the first image set corresponds to a first prompt information in the prompt information set; For each candidate object in the candidate object set, based on the candidate object and the prompt information set, determine a second image set matching the candidate object, wherein the second image set includes at least one second image, each second image corresponds to one of the first prompt information, and the candidate object is used to represent an image style; Based on the attribute information of the first image set and the attribute information of the second image set corresponding to each of the candidate objects, a first object set is determined from the candidate object set.

2. The method according to claim 1, wherein: The determining of the first object set from the candidate object set based on the attribute information of the first image set and the attribute information of the second image set corresponding to each of the candidate objects comprises: For each candidate object in the candidate object set, based on the attribute information of the first image set and the attribute information of the second image set corresponding to the candidate object, determine the detection result of the candidate object; wherein the detection result of each candidate object indicates whether the corresponding candidate object needs to be deleted from the candidate object set; Based on the detection result of each of the candidate objects, the first object set is determined from the candidate object set.

3. The method according to claim 2, wherein: The attribute information of the first image set includes a first rating value, and the attribute information of the second image set includes a second rating value; The determining the detection result of the candidate object based on the attribute information of the first image set and the attribute information of the second image set corresponding to the candidate object includes: Determining a first scoring value corresponding to the first image set based on the scoring value of each first image in the first image set; Determining a second scoring value corresponding to the second image set based on the scoring value of each second image in the second image set; A detection result of the candidate object is determined based on the first scoring value and the second scoring value.

4. The method according to claim 3, wherein: The determining, based on the first scoring value and the second scoring value, a detection result of the candidate object comprises at least one of the following: When the second score value is less than the first score value, taking the first detection result as the detection result of the candidate object; wherein the first detection result indicates that the candidate object needs to be deleted from the candidate object set; When the second score value is not less than the first score value, the second detection result is used as the detection result of the candidate object; wherein the second detection result indicates that the candidate object does not need to be deleted from the candidate object set.

5. The method according to claim 2, wherein: The attribute information of the first image set includes a first cross-modal similarity, and the attribute information of the second image set includes a second cross-modal similarity; The determining the detection result of the candidate object based on the attribute information of the first image set and the attribute information of the second image set corresponding to the candidate object includes: Determining a first cross-modal similarity corresponding to the first image set based on each first image in the first image set and first prompt information corresponding to each first image; Determine a second cross-modal similarity corresponding to the second image set based on each second image in the second image set and the first prompt information corresponding to each second image; A detection result of the candidate object is determined based on the first cross-modal similarity and the second cross-modal similarity.

6. The method according to claim 5, wherein: The determining, based on each first image in the first image set and the first prompt information corresponding to each first image, a first cross-modal similarity corresponding to the first image set includes: For each first image in the first image set, determining a first similarity between the first image and first prompt information corresponding to the first image; Based on each of the first similarities, determining a first cross-modal similarity corresponding to the first image set; The determining, based on each second image in the second image set and the first prompt information corresponding to each second image, a second cross-modal similarity corresponding to the second image set includes: For each second image in the second image set, determining a second similarity between the second image and the first prompt information corresponding to the second image; Based on each of the second similarities, a second cross-modal similarity corresponding to the second image set is determined.

7. The method according to claim 5, wherein: The determining, based on the first cross-modal similarity and the second cross-modal similarity, a detection result of the candidate object includes: Determining a third cross-modal similarity based on the first cross-modal similarity and the second cross-modal similarity; When the third cross-modal similarity is less than a similarity threshold, using the first detection result as a detection result of the candidate object; When the third cross-modal similarity is not less than the similarity threshold, the second detection result is used as the detection result of the candidate object.

8. The method according to claim 2, wherein: The step of determining the first object set from the candidate object set based on the detection result of each candidate object comprises at least one of the following: In a case where the detection result of each of the candidate objects is the second detection result, taking the candidate object set as the first object set; In a case where the detection results of each of the candidate objects include at least one first detection result, deleting the candidate objects corresponding to each of the first detection results from the candidate object set, and using the deleted candidate object set as a new candidate object set; Determining a new first image set based on the new prompt information set; For each candidate object in the new candidate object set, based on the new prompt information set and the candidate object, determine a new second image set matching the candidate object; Based on the attribute information of the new first image set and the attribute information of the new second image set corresponding to each of the candidate objects, the first object set is determined from the new candidate object set.

9. The method according to any one of claims 1 to 8, wherein: The step of determining a second image set matching the candidate object based on the candidate object and the prompt information set includes: For each first prompt information in the prompt information set, second prompt information is determined based on the candidate object and the first prompt information, and a second image matching the candidate object is determined based on the second prompt information.

10. A method for generating an image, the method comprising: Determine at least one target object from a second object set; wherein the second object set is obtained according to the method of any one of claims 1 to 9; Based on the prompt information and each of the target objects, an image corresponding to the prompt information is generated.

11. An object set generation device, the device comprising: The first determining part is configured to determine a first image set based on the prompt information set; wherein each first image in the first image set corresponds to a first prompt information in the prompt information set; A second determining part is configured to determine, for each candidate object in the candidate object set, a second image set matching the candidate object based on the candidate object and the prompt information set, wherein the second image set includes at least one second image, each second image corresponds to one of the first prompt information, and the candidate object is used to represent the image style; The first generating part is configured to determine a first object set from the candidate object set based on the attribute information of the first image set and the attribute information of the second image set corresponding to each of the candidate objects.

12. An image generating device, the device comprising: A third determining part is configured to determine at least one target object from a second object set; wherein the second object set is obtained according to the method of any one of claims 1 to 9; The second generating part is configured to generate an image corresponding to the prompt information based on the prompt information and each of the target objects.

13. An electronic device comprising a processor and a memory, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 10 is implemented.

14. A computer-readable storage medium, wherein: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

15. A computer program comprising computer readable code, wherein: When the computer readable code is executed in a computer device, a processor in the computer device executes the steps for implementing the method according to any one of claims 1 to 10.

16. A computer program product, wherein: The computer program product comprises a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, the steps in the method according to any one of claims 1 to 10 are implemented.

Images