Image generation method and apparatus

Abstract

Embodiments of the present specification provide an image generation method and device, the method comprising: determining a candidate image generation model set from an initial image generation model set according to a target generation style; scoring each candidate image generation model to obtain a scoring result, and determining a target image generation model from the candidate image generation model set according to the scoring result; inputting an initial reference image containing a reference object and an object generation prompt word corresponding to the target image generation model into the target image generation model, to generate at least one target reference image containing the reference object, the object generation prompt word containing multiple generation dimensions. The object generation prompt word containing multiple generation dimensions guides the target image generation model to expand richer object information from the limited initial reference image, to generate at least one target reference image of more dimensions, thereby effectively improving the generation effect when visual content generation is performed using at least one reference image.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to an image generation method. One or more embodiments of this specification also relate to an image generation apparatus, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Currently, in video generation, the reference images used to define characters typically include three views: front, side, and back. However, these reference images can only present limited character information. Furthermore, when generating these reference images, different generation models have varying abilities to adapt to different generation styles and character features. This adaptation difference can lead to inconsistent character appearances and style mismatches, which in turn affects the subsequent generation results. Summary of the Invention

[0003] In view of this, embodiments of this specification provide an image generation method. One or more embodiments of this specification also relate to an image generation apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical defects in the prior art where the limitations of the reference image lead to poor generation results.

[0004] According to a first aspect of the embodiments of this specification, an image generation method is provided, comprising: A set of candidate image generation models is determined from the initial set of image generation models based on the target generation style; Each candidate image generation model in the candidate image generation model set is scored to obtain a score result, and a target image generation model is determined from the candidate image generation model set based on the score result. An initial reference image containing the reference object and object generation prompts corresponding to the target image generation model are input into the target image generation model to generate at least one target reference image containing the reference object and the target generation style. The object generation prompts contain descriptive text describing the reference object in multiple generation dimensions.

[0005] According to a second aspect of the embodiments of this specification, an image generation apparatus is provided, comprising: The candidate model determination module is configured to determine a set of candidate image generation models from the initial set of image generation models based on the target generation style; The target model determination module is configured to score each candidate image generation model in the candidate image generation model set, obtain a score result, and determine a target image generation model from the candidate image generation model set based on the score result. The image generation module is configured to input an initial reference image containing a reference object and an object generation prompt corresponding to the target image generation model into the target image generation model, and generate at least one target reference image containing the reference object and the target generation style, wherein the object generation prompt contains descriptive text describing the reference object in multiple generation dimensions.

[0006] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described image generation method.

[0007] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the image generation method described above.

[0008] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the image generation method described above.

[0009] This specification provides an embodiment of an image generation method. A set of candidate image generation models is determined from an initial set of image generation models based on a target generation style. This ensures that model selection is adapted to specific generation style requirements, laying the foundation for accurate determination of the target image generation model. Each candidate image generation model in the set is scored to obtain a score result, and the target image generation model is determined from the set based on the score result. The scoring mechanism objectively quantifies the generation capability of each candidate image generation model, thereby selecting a target image generation model that is well-suited to the current generation task. An initial reference image containing a reference object and object generation prompts corresponding to the target image generation model are input into the target image generation model to generate at least one target reference image containing the reference object and the target generation style. The object generation prompts contain descriptive text describing the reference object across multiple generation dimensions. These prompts guide the target image generation model to expand richer object information from the limited initial reference image, generating at least one target reference image that is consistent with the reference object across multiple generation dimensions. This effectively improves the appearance consistency of the reference object and enhances the generation effect when using at least one reference image for corresponding visual content generation. Attached Figure Description

[0010] Figure 1This is a scene diagram illustrating an image generation method provided in one embodiment of this specification; Figure 2 This is a flowchart of an image generation method provided in one embodiment of this specification; Figure 3 This is a flowchart illustrating the processing steps of an image generation method provided in one embodiment of this specification. Figure 4 This is a schematic diagram of the structure of an image generation apparatus provided in one embodiment of this specification; Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0011] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0012] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0013] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0014] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0015] Main image: A reference image of the target character uploaded by the user or generated by the system.

[0016] Three-view drawings: Reference drawings including the front, side, and back views of the character.

[0017] Character Panel: A collection of character reference images or images containing multiple perspectives, expressions, poses, and corresponding outfits.

[0018] Model capability information: Data representing the capabilities of different image generation models in terms of specific style, character consistency, multi-view integrity, detail representation, and human proportions.

[0019] Board-level reference: Users reference multidimensional character boards (or reference images related to target shots) as reference inputs to the video generation model in storyboarding or video generation.

[0020] This specification provides an image generation method, and also relates to an image generation apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0021] See Figure 1 , Figure 1 A schematic diagram of a scene illustrating an image generation method provided in one embodiment of this specification is shown.

[0022] Specifically, the image generation method is applied to an image generation system, which includes an end-side device 102 and a server 104. The end-side device 102 is used to send the target generation style to the server 104, such as the target generation style being realistic, ancient costume, 3D animation, 2D animation, etc.

[0023] Server 104 contains multiple image generation models, which together form the initial image generation model set. The image generation models can be third-party models called through the model interface, self-developed models, or privately deployed models. This manual does not limit the specific source or training method of the models.

[0024] Server 104 determines a set of candidate image generation models from the initial set of image generation models based on the target generation style; it scores each candidate image generation model in the set of candidate image generation models to obtain a score result, and determines a target image generation model from the set of candidate image generation models based on the score result; it inputs an initial reference image containing a reference object and an object generation prompt corresponding to the target image generation model into the target image generation model to generate at least one target reference image containing the reference object and the target generation style, wherein the object generation prompt contains descriptive text describing the reference object in multiple generation dimensions. Server 104 can return the generated at least one target reference image to the edge device 102.

[0025] The edge device 102 may include a browser, an app (application), or a web application such as an H5 (Hypertext Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. The edge device can be developed based on a software development kit (SDK) provided by the server, such as a real-time communication (RTC) SDK. The edge device can be deployed in an electronic device and depends on the device's operation or certain apps within the device to run. The electronic device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured in the electronic device, such as human-computer interaction applications, model training applications, image generation applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0026] Server 104 can be understood as a server providing various services, including physical servers and cloud servers. Examples include servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It's important to note that Server 104 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. Server 104 can also be a server in a distributed system, or a server integrated with blockchain. Server 104 can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0027] It is worth noting that the image generation method provided in the embodiments of this specification can be executed by the server 104. In other embodiments of this specification, the image generation model can be deployed in the end-side device 102, so that the end-side device 102 can also have similar functions to the server 104, thereby executing the image generation method provided in the embodiments of this specification. In other embodiments, the image generation method provided in the embodiments of this specification can also be jointly executed by the end-side device 102 and the server 104.

[0028] See Figure 2 , Figure 2A flowchart of an image generation method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0029] Step 202: Determine the candidate image generation model set from the initial image generation model set according to the target generation style.

[0030] The target generation style can be understood as the visual style selected by the user, used to determine the visual style of at least one reference image to be generated. In this embodiment, the target generation style can be any one of live-action, live-action ancient costume, 3D animation, 2D animation, or ink-wash martial arts, without any limitation. The initial image generation model set can be understood as a set of selectable image generation models. The candidate image generation model set can be understood as a set of image generation models selected from the initial image generation model set that have the ability to match the target generation style.

[0031] In practical applications, the choice of image generation model directly affects the quality of the generated image. Different image generation models produce different results for different visual styles. If a fixed image generation model is used directly to generate images with different visual styles, problems such as poor style adaptation and unstable generation quality may occur. Therefore, it is necessary to determine a set of candidate image generation models from the initial set of image generation models based on the target generation style to narrow down the selection range and provide a basis for subsequent scoring.

[0032] Specifically, for the provided initial image generation model set, the models suitable for generating corresponding images under the target generation style are initially selected based on the target generation style, forming a candidate image generation model set. Subsequently, each candidate image generation model in the candidate image generation model set is scored, without having to score all the initial image generation models one by one, which reduces the amount of data processing and saves computing resources.

[0033] It should be noted that the generation styles that an image generation model can adapt to can be determined by preset configuration, or the generation styles that the image generation model can adapt to can be dynamically determined based on the generation effect of the image generation model in various generation styles. No limitation is made here.

[0034] For example, when the target generation style is realistic, image generation models a and b, which are good at realistic style, have good object consistency, and have better human proportions, can be selected from the initial image generation model set as candidate image generation model sets; when the target generation style is 3D animation, image generation models c and d, which are good at 3D animation style, have high multi-view integrity, and have good detail, can be selected from the initial image generation model set as candidate image generation model sets.

[0035] By determining the candidate image generation model set from the initial image generation model set based on the target generation style, subsequent model scoring and model determination can be carried out within a range that is more suitable for the target generation style, reducing the problem of poor image generation results caused by using the same image generation model for image generation and being unable to adapt to different styles.

[0036] In one or more embodiments of this specification, the initial image generation model set may contain a large number of models, and different models are good at generating different styles. Therefore, it is necessary to accurately select the candidate image generation model set based on the model capability information of each initial image generation model and the target generation style. Specific implementation methods are described below: The step of determining the candidate image generation model set from the initial image generation model set according to the target generation style includes: Obtain the model capability information of each initial image generation model in the initial image generation model set, wherein the model capability information includes the style adaptability of the initial image generation model to each generation style; Based on the target generation style and the model capability information of each initial image generation model, models whose style fit with the target generation style reaches a preset fit threshold are selected from the initial image generation model set to obtain the candidate image generation model set.

[0037] Among them, model capability information can be understood as data representing the initial image generation model's generation capabilities in a specific generation style, object consistency, multi-view integrity, detail representation, and human body proportions; style fit is determined based on information such as the quality level, consistency, multi-view integrity, detail representation, or human body proportions of the images generated by the image generation model under a specific generation style.

[0038] The preset adaptation threshold can be understood as a pre-set criterion used to measure the degree of adaptation between the initial image generation model and the target generation style. When the style adaptation degree between the initial image generation model and the target generation style reaches the preset adaptation threshold, it is determined that the initial image generation model has the ability to generate the target generation style, and it is used as a candidate image generation model. When the style adaptation degree does not reach the preset adaptation threshold, it is determined that the initial image generation model does not have the ability to generate the target generation style. It should be noted that the preset adaptation threshold is used for preliminary screening of the initial image generation model, and its specific value can be set according to the target generation style, the form of model capability information, and actual application requirements. This embodiment does not limit this.

[0039] In practical applications, since the model capability information includes the style fit of the initial image generation model to each generation style, it is possible to determine a relatively suitable candidate image generation model for different generation styles.

[0040] Specifically, the model capability information corresponding to each initial image generation model in the initial image generation model set is obtained. This model capability information records the style adaptability of each initial image generation model to different generation styles. The target generation style is compared with the model capability information of each initial image generation model, and models whose style adaptability to the target generation style reaches a preset adaptation threshold are selected. These models are then combined into a candidate image generation model set, meaning that all models in the candidate image generation model set are adapted to the target generation style.

[0041] For example, the initial image generation model set includes models A, B, and C. Model A's capability information indicates that it has strong generation capabilities in realistic style but weak capabilities in 3D animation style; model B's capability information indicates that it has strong generation capabilities in 3D animation style but weak capabilities in realistic style; and model C's capability information indicates that it has strong generation capabilities in ink painting style. When the target generation style is 3D animation, model B can be selected from the initial image generation model set as the model with the capability to generate the target style, based on the target generation style and the capability information of each initial image generation model, thus obtaining a candidate image generation model set containing model B.

[0042] The image generation method provided in the embodiments of this specification filters out models with the ability to generate the target style based on the model capability information of each initial image generation model, and obtains a set of candidate image generation models. This makes the determination process of the candidate image generation model set more accurate and efficient, and avoids including models that do not have the ability to generate the target style in the subsequent model scoring process.

[0043] Step 204: Score each candidate image generation model in the candidate image generation model set to obtain the score result, and determine the target image generation model from the candidate image generation model set based on the score result.

[0044] The scoring can be understood as the process of quantifying the expected quality of the candidate image generation model in generating the corresponding image under the target generation style; the scoring result can be understood as the quantification result of the expected generation quality of the candidate image generation model under the target generation style; the target image generation model can be understood as the model finally selected from the set of candidate image generation models to generate at least one target reference image containing the reference character, which can also be called the relatively better model.

[0045] Specifically, for each candidate image generation model in the candidate image generation model set, a score is given according to the same evaluation criterion to obtain the score result corresponding to each candidate image generation model. Then, based on the score result, these candidate image generation models are compared, sorted or filtered to determine one or more target image generation models, so that subsequent image generation is implemented by the target image generation model.

[0046] In practice, since the target image generation model is determined based on the scoring results among candidate image generation models that have the ability to generate the target image style, when the target image generation model and its matching prompt words are subsequently called to generate reference images, a reference image of relatively higher quality can be obtained.

[0047] For example, the candidate image generation model set includes model A, model B, and model C, and the target generation style is 3D animation. The expected quality of the corresponding images generated by model A, model B, and model C in the 3D animation style can be scored respectively, and the scores are obtained as follows: model A is 85 points, model B is 92 points, and model C is 78 points. Based on the score results, model B is determined as the target image generation model.

[0048] By scoring each candidate image generation model in the candidate image generation model set, and then determining the target image generation model, it is possible to further select a more suitable model from the candidate models that have been adapted to the target generation style, thus providing a basis for improving the generation quality of the reference image.

[0049] In one or more embodiments of this specification, although each candidate image generation model in the candidate image generation model set has the ability to generate the target generation style, different candidate image generation models may have differences in different capabilities. Therefore, based on the scoring dimension corresponding to the target generation style and the dimensional weight of each scoring dimension, combined with the model capability information of each candidate image generation model, the true fit of each candidate image generation model under the target generation style can be obtained more accurately, and the interpretability of the scoring results can be improved.

[0050] The step of scoring each candidate image generation model in the candidate image generation model set to obtain a scoring result includes: Determine at least one rating dimension corresponding to the target generation style and the dimensional weight of each rating dimension; Based on the model capability information of each candidate image generation model in the candidate image generation model set, determine the dimension score of each candidate image generation model in each scoring dimension; The scoring results of each candidate image generation model are obtained based on the dimensional scores of each candidate image generation model in each scoring dimension and the dimensional weights of each scoring dimension.

[0051] The scoring dimension can be understood as an angle or indicator used to evaluate the expected quality of the corresponding image generated by the candidate image generation model under the target generation style. For example, the scoring dimension includes at least one of the following: style matching degree (the degree of matching between the model and the target generation style), object consistency (the ability of the model to maintain the object's identity when generating different views), multi-view integrity (the degree of integrity of the multi-dimensional view generated by the model), detail representation (the quality of details such as face, hairstyle, clothing and accessories generated by the model), and human proportion (the reasonableness of the human proportion generated by the model).

[0052] Of course, in practical applications, the scoring dimension can include not only dimensions related to generation quality, but also dimensions related to generation efficiency or generation cost. For example, in some embodiments, the scoring dimension corresponding to each generation style also includes dimensions such as generation efficiency and generation cost. When the target image generation model is determined through multiple scoring dimensions, a better balance between quality, efficiency and cost can be achieved based on the determined target image generation model.

[0053] Dimension weights can be understood as the proportion or importance of each scoring dimension in the scoring result. The dimensional weights corresponding to different target generation styles can be the same or different, which is not limited here; dimensional scores can be understood as the score of the candidate image generation model on a certain scoring dimension.

[0054] Specifically, first, determine at least one scoring dimension corresponding to the target generation style and the dimensional weight of each scoring dimension. Then, based on the model capability information of each candidate image generation model in the candidate image generation model set, determine the dimensional score of each candidate image generation model on each scoring dimension. Finally, calculate the dimensional scores of each candidate image generation model on each scoring dimension according to the dimensional weight of each scoring dimension to obtain the scoring result of each candidate image generation model.

[0055] In practical applications, different generation styles correspond to different generation configuration information. When the target generation style is determined, the target generation configuration information corresponding to the target generation style is determined. The target generation configuration information includes a set of candidate image generation models that can be called, model selection rules (including at least one scoring dimension and the dimensional weight of each scoring dimension), at least one generation dimension, etc.

[0056] For example, if the target generation style is realistic portraiture, the scoring dimensions can be determined to include object consistency, human proportion, and detail representation, with the weights for object consistency (0.5), human proportion (0.3), and detail representation (0.2). Based on the model capability information of models A, B, and C in the candidate image generation model set, model A is determined to score 90 points in object consistency, 80 points in human proportion, and 85 points in detail representation; model B is determined to score 85 points in object consistency, 90 points in human proportion, and 80 points in detail representation; and model C is determined to score 80 points in object consistency, 85 points in human proportion, and 90 points in detail representation. Weighted calculations are then performed according to the weights of each scoring dimension to obtain the individual scores for models A, B, and C.

[0057] The image generation method provided in the embodiments of this specification, by determining at least one scoring dimension corresponding to the target generation style and the dimensional weight of each scoring dimension, can make the scoring results more in line with the actual needs of the target generation style, and improve the accuracy and rationality of the comparison of candidate image generation models.

[0058] In one or more embodiments of this specification, after obtaining the scoring results of each candidate image generation model, one or more target image generation models can be determined from the candidate image generation model set based on the scoring results in multiple ways, thereby achieving flexible determination of the target image generation model. Specific implementation methods are described below: The step of determining the target image generation model from the candidate image generation model set based on the scoring result includes: Based on the scoring results, each candidate image generation model in the candidate image generation model set is sorted in descending order to generate a sorted list, and the candidate image generation model that is first in the sorted list is determined as the target image generation model. Alternatively, the candidate image generation models in the candidate image generation model set can be sorted in descending order according to the scoring results to generate a sorted list. The sorted list can be returned to the client, and the target image generation model can be determined from the sorted list in response to the client's selection operation in the sorted list. Alternatively, the candidate image generation model in the candidate image generation model set whose scoring results exceed a preset scoring threshold can be determined as the target image generation model.

[0059] The descending sorting can be understood as arranging the candidate image generation models from highest to lowest score; the sorted list can be understood as the list of candidate image generation models obtained by sorting in descending order, with the candidate image generation model at the top of the sorted list being the candidate image generation model with the highest score; the selection operation can be understood as the operation of the client user clicking, checking, confirming, or specifying a candidate image generation model in the sorted list; and the preset score threshold can be understood as a pre-set score limit used to determine whether the candidate image generation model meets the requirements of the target image generation model.

[0060] Specifically, determining the target image generation model from the candidate image generation model set based on the scoring results can be implemented in the following ways: The first way is to automatically select the model with the higher score, that is, to sort the candidate image generation models in the candidate image generation model set in descending order according to the scoring results, generate a sorted list, and determine the candidate image generation model that is first in the sorted list as the target image generation model; The second way is to have the user participate in the selection, that is, to return the sorted list to the client, and in response to the client's selection operation in the sorted list, to take the model determined by the user from the sorted list as the target image generation model; The third way is threshold screening, that is, to determine the candidate image generation model in the candidate image generation model set whose scoring results exceed a preset scoring threshold as the target image generation model.

[0061] In practical applications, there may be one or more candidate image generation models in the candidate image generation model set whose scores exceed a preset scoring threshold; that is, there may be one or more target image generation models. When there are multiple target image generation models, an initial reference image containing the reference object and object generation prompts corresponding to each target image generation model can be input into each model. Each model will then generate at least one target reference image containing the reference object and the target generation style. Each target image generation model can generate one or more target reference images; there is no limitation on this.

[0062] In one possible implementation, when there are multiple target image generation models, a corresponding number of generation can be set for each target image generation model. The initial reference image and the corresponding object generation prompt are input into each target image generation model, and each target image generation model generates a target reference image according to the corresponding number of generation, thereby obtaining at least one target reference image jointly generated by multiple target image generation models.

[0063] In another possible implementation, when there are multiple target image generation models, the target reference images generated by each target image generation model can be aggregated and filtered or sorted according to the consistency of appearance between each target reference image and the reference object and / or the matching degree with the target generation style, so as to determine the final target reference image used for subsequent visual content generation.

[0064] In other words, regardless of whether there is one or more target image generation models, one or more target reference images can be generated. When there are multiple target image generation models, at least one target reference image can be generated by one of the target image generation models, or at least one target reference image can be generated by multiple target image generation models respectively. This embodiment does not limit this.

[0065] For example, if the target image generation style is realistic portraiture, and the candidate image generation model set includes Model A, Model B, and Model C, with scores of 92, 88, and 80 respectively, the candidate image generation models can be sorted in descending order based on their scores, generating a sorted list of Model A, Model B, and Model C. Model A, ranked first in the list, can then be identified as the target image generation model. Alternatively, the sorted list can be returned to the client, allowing the user to select Model B. In response to this selection, Model B can be identified as the target image generation model. Furthermore, a preset scoring threshold of 85 points can be set, identifying Model A and Model B, whose scores exceed 85 points, as the target image generation model.

[0066] In practical applications, different selection methods can be applied to different scenarios. For example, automatic model selection is suitable for scenarios that pursue efficiency, user participation selection is suitable for scenarios that require manual control of image generation quality, and threshold screening is suitable for scenarios that allow multiple models to generate multiple image candidates.

[0067] The image generation method provided in the embodiments of this specification can take into account various application requirements such as automatic determination, user-participatory determination, and threshold determination by determining the target image generation model in multiple ways, thereby improving the flexibility and accuracy of the target image generation model determination.

[0068] Step 206: Input the initial reference image containing the reference object and the object generation prompt words corresponding to the target image generation model into the target image generation model to generate at least one target reference image containing the reference object and the target generation style, wherein the object generation prompt words contain descriptive text describing the reference object in multiple generation dimensions.

[0069] The reference object can be understood as the object to be presented in the generated image. This reference object can be a person, a subject, or other objects that need to maintain a consistent appearance; there are no specific limitations. The initial reference image can be understood as an image containing the reference object, serving as the basis for generation, such as a user-uploaded subject image or a subject image generated by the system based on the description of the reference object. It should be noted that the initial reference image primarily presents the appearance features of the reference object. If the initial reference image also contains a background or other objects, the target image generation model can generate at least one target reference image containing the reference object, guided by multiple generation dimensions in the object generation prompts.

[0070] Object generation prompts can be understood as prompts corresponding to the target image generation model, used to guide the target image generation model to generate images related to the reference object. The object generation prompts may contain object description information of the reference object and description information of the scene or environment in which the reference object is located. The object generation prompts contain descriptive text describing the reference object in multiple generation dimensions.

[0071] The generation dimension can be understood as the different aspects in the object generation prompts used to constrain or guide the generation direction, such as perspective dimension, expression dimension, pose dimension, clothing dimension, hairstyle dimension, accessory dimension, or human proportion dimension. The target reference image can be understood as the reference image output by the target image generation model that contains the reference object, and the number of such reference images can be one or more.

[0072] Specifically, given a target image generation model, the object generation prompt corresponding to the target image generation model is invoked. The initial reference image containing the reference object and the object generation prompt are input into the target image generation model. The target image generation model generates one or more target reference images containing the reference object based on the appearance of the reference object in the initial reference image and the guidance of multiple generation dimensions in the object generation prompt. This allows the generated target reference images to cover the content indicated by multiple generation dimensions.

[0073] It should be noted that the initial reference image containing the reference object is used to provide an appearance reference for the target image generation model. Therefore, the initial reference image is usually an image that contains a relatively comprehensive set of appearance features of the reference object. For example, if the reference object is a person, the initial reference image can be a frontal main image of the person.

[0074] For example, when the target generation style is 3D animation, an initial reference image containing the reference character and object generation prompts corresponding to the 3D animation style can be input into the target image generation model. The object generation prompts contain descriptive text describing the reference object in the perspective dimension, expression dimension, and posture dimension. Therefore, the target image generation model can generate multiple target reference images, such as a frontal view image, a three-quarter view image, a smiling expression image, and a combat posture image of the reference character.

[0075] In this embodiment, the object generation prompts corresponding to the target image generation model contain descriptive text describing the reference object in multiple generation dimensions, thereby enabling the generated target reference image to cover object information in multiple dimensions, enriching the image of the reference object in different dimensions, and providing rich visual information for the subsequent generation of visual data of the reference object.

[0076] In one or more embodiments of this specification, before inputting the initial reference image and object generation prompts into the target image generation model, the object generation prompts corresponding to the target image generation model are invoked. These object generation prompts contain multiple generation dimensions and are constructed based on the dimension prompts for each generation dimension. Therefore, a target reference image covering multiple dimensions can be generated based on the object generation prompts. Specific implementation methods are as follows: Before inputting an initial reference image containing the reference object and an object generation prompt word corresponding to the target image generation model into the target image generation model to generate at least one target reference image containing the reference object and the target generation style, the method further includes: The object generation prompts corresponding to the target image generation model are invoked, wherein the object generation prompts include dimension prompts for at least two of the generation dimensions: viewpoint dimension, expression dimension, posture dimension, and clothing dimension.

[0077] Among them, the object generation prompt words corresponding to the target image generation model can be understood as prompt words adapted to the target image generation model and used to guide the target image generation model to generate reference object related images; the viewpoint dimension can be understood as the dimension used to guide the generation of different observation angles, such as front view, side view, back view, three-quarter view, top view, or bottom view; the expression dimension can be understood as the dimension used to guide the generation of different facial expressions, such as smiling, sad, angry, or surprised; the posture dimension can be understood as the dimension used to guide the generation of different body movements or postures, such as standing, walking, fighting, or sitting; the clothing dimension can be understood as the dimension used to guide the generation of different clothing, accessories, or wearing states; the dimension prompt words can be understood as prompt words set for a certain generation dimension and used to specifically guide the generation of content in that dimension. The object generation prompt words can be constructed based on the dimension prompt words of each generation dimension.

[0078] Specifically, before inputting the initial reference image containing the reference object and the object generation prompt words corresponding to the target image generation model into the target image generation model, the object generation prompt words corresponding to the target image generation model can be called first. These object generation prompt words are not a single general description, but rather contain dimensional prompt words that include at least two of the generation dimensions, such as viewpoint dimension, expression dimension, posture dimension, and clothing dimension. This allows the called object generation prompt words to guide the generation process of the target image generation model from multiple dimensions.

[0079] For example, if the target image generation style is realistic, the object generation prompt words corresponding to the target image generation model under the realistic style can be called. These object generation prompt words include dimensional prompt words for the perspective dimension, expression dimension, and clothing dimension. The perspective dimension prompt words are used to guide the generation of front, side, back, three-quarter view, top view, and bottom view perspectives. The expression dimension prompt words are used to guide the generation of expressions such as smiling, sad, and angry. The clothing dimension prompt words are used to guide the generation of different clothing and accessory states.

[0080] The image generation method provided in the embodiments of this specification calls the object generation prompt corresponding to the target image generation model before inputting the initial reference image and object generation prompt words into the target image generation model. This enables the object generation prompt words to match the target image generation model and the target generation style, and guides the generation of the target reference image from multiple generation dimensions contained in the object generation prompt words. This provides guidance support for generating target reference images that cover multiple perspectives, multiple expressions, multiple poses and clothing details.

[0081] In one or more embodiments of this specification, after the target image generation model generates at least one target reference image, it sends the target reference image to the client. Upon receiving a confirmation response, the at least one target reference image is identified as an object reference atlas of the reference object and stored in the object library, so that the confirmed object reference atlas can provide a reference for generating target visual data of the reference object. Specific implementation methods are described below: After inputting an initial reference image containing the reference object and an object generation prompt word corresponding to the target image generation model into the target image generation model to generate at least one target reference image containing the reference object and the target generation style, the method further includes: In response to the client's confirmation operation of the at least one target reference image, an object reference atlas of the reference object is determined based on the at least one target reference image, and the object reference atlas of the reference object is stored in an object library, wherein the object reference atlas is used to provide a reference for generating target visual data of the reference object.

[0082] Among them, the object reference atlas can be understood as a collection of at least one target reference image that is used as a reference basis for reference objects after being confirmed by the client, and can also be called a multi-dimensional object board; the object library can be understood as an asset library used to store the object reference atlas related to the reference object, which contains object reference atlases of different reference objects; the target visual data can be understood as the visual data containing the reference object that needs to be generated later, such as storyboards, video clips, etc.

[0083] Specifically, after inputting the initial reference image containing the reference object and the object generation prompt words corresponding to the target image generation model into the target image generation model, and generating at least one target reference image containing the reference object, the generated at least one target reference image can be sent to the client for the user to confirm and select from at least one target reference image.

[0084] In response to the client's confirmation operation of the at least one target reference image, an object reference atlas of the reference object is determined based on the at least one target reference image, and the object reference atlas of the reference object is stored in the object library, thereby making the object reference atlas an object asset that has been confirmed by the user and can be called later, and used to provide a reference for generating target visual data of the reference object.

[0085] In practical applications, by storing the object reference atlas in the object library, the object reference atlas can be reused in scenarios such as storyboard generation and image-generated video, thereby improving the reuse efficiency of object assets.

[0086] For example, after the target image generation model generates multiple target reference images containing a reference character, the client user can view and confirm these target reference images. Upon user confirmation, the object reference atlas corresponding to the reference character is determined based on these target reference images or a subset of the user-confirmed target reference images, and stored in an object library. When subsequently generating storyboards or video clips for the reference character, this object reference atlas can be retrieved from the object library as a visual reference.

[0087] The image generation method provided in the embodiments of this specification determines an object reference atlas based on at least one target reference image and stores it in an object library. This enables subsequent storyboards and video generation to reference the user-confirmed object reference atlas. The object reference atlas contains more dimensions of object information, thus reducing problems such as face swapping, hairstyle changes, clothing changes, and body proportion changes in subsequent videos, and providing a foundation for improving the consistency of characters across frames, perspectives, and shots.

[0088] In one or more embodiments of this specification, after the target image generation model generates at least one target reference image, the target reference image needs to be actually used for subsequent visual data generation. During subsequent visual data generation, based on the received visual data generation request for the reference object, generation constraint information is obtained, and the generation constraint information and at least one target reference image are input into the visual data generation model to obtain target visual data. Specific implementation methods are described below: After inputting an initial reference image containing the reference object and an object generation prompt word corresponding to the target image generation model into the target image generation model to generate at least one target reference image containing the reference object and the target generation style, the method further includes: Receive a visual data generation request for the reference object, and obtain generation constraint information from the visual data generation request; The generation constraint information and at least one target reference image containing the reference object are input into the visual data generation model to obtain target visual data.

[0089] Here, "visual data generation request" can be understood as a request to generate visual data related to a reference object; "generation constraint information" can be understood as information carried in the visual data generation request that constrains the direction of target visual data generation, such as generation perspective, object expression, object posture, clothing information, camera information, scene information, or style information, etc., without limitation here; "visual data generation model" can be understood as a model used to generate target visual data, such as a storyboard generation model, image-to-video model, video generation model, motion comic generation model, etc., without limitation here; and "target visual data" can be understood as the visual data containing the reference object that is finally generated by the visual data generation model, such as storyboards, video clips, motion comics, etc.

[0090] Specifically, after generating at least one target reference image containing a reference object, a visual data generation request for the reference object can be received, and generation constraint information can be obtained from the visual data generation request. The generation constraint information and the at least one target reference image containing the reference object are input into the visual data generation model. The visual data generation model generates target visual data based on the generation constraint information and the target reference image, so that the target visual data not only meets the generation requirements defined by the generation constraint information, but also references the appearance of the reference object in the target reference image.

[0091] For example, upon receiving a storyboard generation request for a reference character, the model obtains the camera angle, character expression, and scene information as generation constraints. These constraints, along with multiple target reference images containing the reference character, are then input into the storyboard generation model to obtain a storyboard containing the reference character. As another example, upon receiving a video generation request for a reference character, the model obtains the camera movement, character pose, and clothing information as generation constraints. These constraints, along with multiple target reference images containing the reference character, are then input into a video generation model to obtain a video clip containing the reference character.

[0092] The image generation method provided in the embodiments of this specification obtains target visual data by generating constraint information and at least one target reference image containing a reference object. This enables the generation of target visual data to be constrained by both the generation constraint information and the target reference image, reducing problems such as face swapping, hairstyle changes, clothing changes, and body proportion changes in subsequent videos, and improving the generation quality of target visual data.

[0093] In one or more embodiments of this specification, at least one target reference image containing a reference object may cover multiple viewpoints, multiple expressions, multiple poses, and various clothing. Meanwhile, the generation constraint information in the visual data generation request typically corresponds to a specific shot or a specific generation requirement. Therefore, an associated reference image matching the generation constraint information can be determined from at least one target reference image. The generation constraint information and the associated reference image are then input into the visual data generation model to obtain the target visual data. Specific implementation methods are described below: The step of inputting the generated constraint information and at least one target reference image containing the reference object into the visual data generation model to obtain target visual data includes: From at least one target reference image containing the reference object, determine an associated reference image that matches the generated constraint information; The generated constraint information and the associated reference image are input into the visual data generation model to obtain the target visual data.

[0094] The associated reference image can be understood as a target reference image selected from at least one target reference image containing a reference object that matches the generation constraint information; matching can be understood as the target reference image conforming to the generation requirements indicated by the generation constraint information in terms of viewpoint, expression, posture, clothing, etc.

[0095] Specifically, at least one target reference image containing a reference object is usually not a single image, but a multi-dimensional image collection covering multiple viewpoints, multiple expressions, multiple poses and multiple clothing states. Different target reference images in at least one target reference image respectively carry the appearance features of the reference object under different generation conditions.

[0096] The generation constraint information describes the specific generation conditions that the target visual data to be generated must meet, such as the generation perspective required by the current shot, the expression, posture, and clothing information of the reference object. Therefore, to avoid introducing unnecessary noise by inputting all target reference images into the visual data generation model, this embodiment matches the generation requirements described by the generation constraint information with the appearance conditions carried by each target reference image, selecting one or more target reference images corresponding to the current generation requirements from at least one target reference image as associated reference images.

[0097] The associated reference image can be a single target reference image that completely corresponds to the generation constraint information, or it can be multiple target reference images that correspond to different generation requirements in the generation constraint information. For example, a viewpoint reference image corresponding to the generation viewpoint and an expression reference image corresponding to the object's expression together constitute an associated reference image.

[0098] After determining the associated reference image, constraint information and the associated reference image are input into the visual data generation model. The constraint information is used to limit the generation direction of the target visual data in terms of viewpoint, expression, posture, or clothing. The associated reference image is used to provide the visual data generation model with the appearance of the reference object under the above generation direction. This allows the visual data generation model to generate data directly based on the appearance features already presented in the associated reference image, without having to infer the appearance of the reference object under complex angles, specific expressions, specific postures, or specific clothing. This ensures that the generated target visual data meets the current generation requirements while maintaining the consistency of the appearance of the reference object across viewpoints and lenses.

[0099] For example, if the generated constraint information indicates that the current shot is a three-quarter view and the character's expression is smiling, the three-quarter view image and the smiling expression image can be determined as associated reference images from multiple target reference images containing the reference character. The generated constraint information and the associated reference image are then input into the storyboard generation model to obtain a storyboard containing the reference character.

[0100] The image generation method provided in the embodiments of this specification can determine an associated reference image that matches the generation constraint information from at least one target reference image containing a reference object. This enables the reference image input to the visual data generation model to accurately correspond to the current generation requirements, avoids interference from irrelevant reference information, and reduces the range of object appearance inferences by the visual data generation model by inputting a reference image that is closer to the current generation requirements. This improves appearance consistency and enhances the generation quality of the target visual data.

[0101] In one or more embodiments of this specification, the generated constraint information may contain various types of constraint content. By determining the generated viewpoint, object expression, object posture, and / or clothing information in the generated constraint information, a corresponding viewpoint reference image, expression reference image, posture reference image, and / or clothing reference image are selected as associated reference images from at least one target reference image containing the reference object. Specific implementation methods are described below: Determining an associated reference image matching the generated constraint information from at least one target reference image containing the reference object includes: The dimension constraint information of the target generation dimension in the generation constraint information is determined. From at least one target reference image containing the reference object, at least one candidate reference image of the target generation dimension is determined. And an associated reference image is determined from the at least one candidate reference image according to the dimension constraint information.

[0102] In specific implementation, the generation perspective, object expression, object posture and / or clothing information in the generation constraint information are determined, and the corresponding perspective reference image, expression reference image, posture reference image and / or clothing reference image are selected as the associated reference image from at least one target reference image containing the reference object.

[0103] Among them, the generated viewpoint can be understood as the observation angle that the target visual data needs to present in the viewpoint dimension, as indicated in the generated constraint information, such as front view, side view, back view, three-quarter view, top view, or bottom view; the object expression can be understood as the facial expression that the reference object needs to present in the expression dimension, as indicated in the generated constraint information, such as smiling, sad, angry, or surprised; the object posture can be understood as the body movement or posture that the reference object needs to present in the posture dimension, as indicated in the generated constraint information, such as standing, walking, fighting, or sitting; and the clothing information can be understood as the clothing, accessories, or wearing status that the reference object needs to present in the clothing dimension, as indicated in the generated constraint information.

[0104] A viewpoint reference image can be understood as an image in at least one target reference image that corresponds to a certain viewpoint in the viewpoint dimension; an expression reference image can be understood as an image in the target reference image that corresponds to a certain expression in the expression dimension; a posture reference image can be understood as an image in the target reference image that corresponds to a certain posture in the posture dimension; and a clothing reference image can be understood as an image in the target reference image that corresponds to a certain clothing state in the clothing dimension.

[0105] Specifically, when determining the associated reference image that matches the generated constraint information from at least one target reference image containing the reference object, the generated constraint information can be parsed first to determine the specific constraint items (i.e., dimensional constraint information) contained in the generated constraint information, such as the generation viewpoint, object expression, object pose and / or clothing information.

[0106] After determining the specific constraints, each constraint is used as a selection criterion. At least one target reference image is searched for the target reference image corresponding to each constraint. For example, a viewpoint reference image matching the required generation viewpoint, an expression reference image matching the required object expression, a posture reference image matching the required object posture, and a clothing reference image matching the required clothing information are searched. One or more of the found viewpoint reference images, expression reference images, posture reference images, and / or clothing reference images are selected as associated reference images matching the generation constraint information.

[0107] In practical applications, since the associated reference image corresponds to the generation constraint information in terms of viewpoint, expression, pose and / or clothing, inputting the generation constraint information and the associated reference image into the visual data generation model can yield target visual data that better meets the generation requirements.

[0108] The image generation method provided in the embodiments of this specification achieves accurate searching of matching associated reference images in at least one target reference image by generating specific constraint items in constraint information, thereby improving the accuracy and relevance of referencing target reference images and increasing the efficiency of visual data generation.

[0109] See Figure 3 , Figure 3 A flowchart illustrating the processing steps of an image generation method according to an embodiment of this specification is shown, specifically including the following steps.

[0110] The image generation method provided in the embodiments of this specification can be applied to image creation or video creation scenarios, specifically covering scenarios such as character definition, storyboard generation, image-to-video generation, and dynamic comic generation.

[0111] In practical applications, during the object definition stage of image or video creation, three views containing the front, side, and back views of the reference object are generated. These three views are then used for storyboarding and video generation. However, the three views have problems such as a limited number of perspectives, lack of facial expressions and poses, and lack of details on faces, hairstyles, clothing, and accessories. When complex angles appear in the subsequent video, the model needs to infer the character's appearance on its own, which can easily lead to face swapping, changes in hairstyles, clothing, and body proportions, resulting in poor generation effects.

[0112] Furthermore, different models excel in different styles, and using a single model consistently makes it difficult to guarantee the quality of generated reference images under different visual styles. Therefore, this method, after style selection, can generate multi-dimensional reference images covering multiple viewpoints, expressions, poses, and details, producing relatively higher-quality reference images for users to use in storyboarding and video generation, thereby improving object consistency across frames, viewpoints, and shots.

[0113] Step 302: Determine the visual generation style and the corresponding generation configuration information.

[0114] The visual generation style is the target generation style in the above embodiments. It should be noted that when the user selects a visual generation style, the visual generation style selected by the user is used as the target generation style. However, when the user does not select a visual generation style, the system can determine the visual generation style based on the initial reference image of the reference object.

[0115] The visual generation style can be realistic, period drama, fantasy, vintage, 3D animation, 2D animation, Japanese manga, ink-wash martial arts, or chibi style, etc. The generation configuration information includes candidate model interfaces and corresponding character panel prompts (i.e., object generation prompts for each model). Additionally, user-uploaded or system-generated character reference images can be used as the main image, serving as the initial reference image containing the reference character.

[0116] Therefore, by determining the visual generation style and the corresponding generation configuration information, a style basis and configuration basis are provided for the subsequent selection of candidate image generation models and character board generation, so that character board generation can call the matching model interface and prompt words according to the visual generation style, instead of using a single model.

[0117] Step 304: Filter candidate image generation models.

[0118] Specifically, the model capability information of each initial image generation model in the initial image generation model set is obtained. The model capability information is data representing the capabilities of different image generation models in terms of specific style, character consistency, multi-view integrity, detail representation, and human proportions.

[0119] Based on the target generation style and the model capability information of each initial image generation model, models with the ability to generate in the target style are selected from the initial image generation model set to obtain a candidate image generation model set. In practical applications, the selected candidate image generation models should at least meet the following requirements: support input main image, be able to generate complete character panels or multiple character reference images with consistent appearance, and be able to generate multiple character views as required.

[0120] By selecting candidate image generation models with character template generation capabilities based on style, we avoid including models that do not have the ability to generate the target style in subsequent scoring and invocation, thus providing a model foundation for determining the relatively better model.

[0121] Step 306: Model scoring and determination of the target image generation model.

[0122] Each candidate image generation model in the candidate image generation model set is scored to obtain the score results. Based on the score results, the target image generation model is determined from the candidate image generation model set. The target image generation model is the relatively better model, which is the model with relatively high expected quality for generating character panels of a specified visual generation style within the current candidate model range.

[0123] Specifically, first, determine at least one scoring dimension corresponding to the target generation style and the dimensional weight of each scoring dimension. The scoring dimension may include at least one of style matching degree, character consistency, multi-view integrity, detail representation, and human body proportion. Then, based on the model capability information of each candidate image generation model in the candidate image generation model set, determine the dimensional score of each candidate image generation model in each scoring dimension. Based on the dimensional scores of each candidate image generation model in each scoring dimension and the dimensional weight of each scoring dimension, obtain the scoring result of each candidate image generation model. Finally, determine the target image generation model based on the scoring results.

[0124] Therefore, by further selecting a more suitable target image generation model from the candidate models that have already adapted to the target generation style based on the scoring results, the target image generation model is used to generate the character board of the reference object.

[0125] Step 308: Call the object to generate prompts and generate a multi-dimensional character board.

[0126] Different models correspond to different prompts. The target image is called to generate the corresponding object prompt. The object prompt contains at least two of the following dimensions (also known as reference dimensions): viewpoint dimension, expression dimension, pose dimension, and clothing dimension.

[0127] The perspective dimension may include at least one of the following: frontal perspective, side perspective, back perspective, three-quarters perspective, top-down perspective, and bottom-up perspective; the clothing dimension may include at least one of the following: facial details, hairstyle details, clothing details, accessory details, and material details.

[0128] In practical applications, object generation prompts can include the following: Viewpoint description, used to limit the combination of viewpoints to be generated for this character panel, such as main full-body, neutral full-body, back, side, top-down, bottom-up, three-quarter view, etc.; Posture description, used to limit posture combinations such as standing, sitting, tilting, and squatting; Expression description, used to limit the types of expressions and the range of emotional changes in the portrait and expression study areas; Clothing description, used to limit the detailed study areas of the face, hair, clothing, and accessories. Identity consistency constraints, used to require that each viewpoint and the main image maintain consistency in face, facial proportions, hairstyle, clothing, body proportions, and gesture language; Layout constraints, used to limit solid color backgrounds, separate and non-overlapping viewpoints, no facial cropping, the main viewpoint as the visual anchor point, and auxiliary research arranged around it; Semantic annotation description, used to limit the content of character identifier blocks such as name, role, and core emotion; Downstream usability constraints, used to require clear facial shapes, hairstyle outlines, clothing outlines, body shapes, hands, postures, and expression ranges.

[0129] It should be noted that, based on the above dimensions, the object generation prompts can also be adaptively adjusted according to the type of the target image generation model. For example, for models with layout and text rendering capabilities, complete layout constraints and semantic annotation instructions can be injected, allowing the model to output a complete character panel in one go; for models with strong realism capabilities, corresponding style human proportion constraints (to prevent characters from becoming shorter or having abnormal proportions) and material rendering instructions can be injected, with layout compositing and semantic annotation handled by the system; for multi-image generation models, the above dimensions can be broken down into multiple single-viewpoint or single-pose prompts generated separately, and then the system arranges them according to layout constraints. All types of prompts include common constraints, such as preventing characters from becoming shorter, face swapping, clothing changes, and view repetition and overlap. Different models can use different prompt structures.

[0130] An initial reference image containing the reference object and object generation prompts corresponding to the target image generation model are input into the target image generation model to generate at least one target reference image containing the reference object. Because the object generation prompts contain multiple generation dimensions, the generated target reference image covers multiple perspectives, multiple expressions, multiple poses and / or clothing. Based on at least one target reference image, a multi-dimensional character board of the reference object can be constructed. This multi-dimensional character board is a character reference image or a set of character reference images (i.e., the object reference image set in the above embodiment) that contains multiple perspectives, expressions, poses and clothing information of the character. This overcomes the problems of limited character information in traditional three-view drawings and the lack of references for three-quarter view, top view, bottom view and other angles, as well as expressions, poses and clothing.

[0131] In this embodiment, if the target image generation model fails to generate or the generation result does not meet the preset requirements, the generation can be re-executed, or other candidate image generation models can be determined from the candidate image generation model set and generated.

[0132] Step 310: Client 102 confirms and stores the object reference atlas.

[0133] In practical applications, when the client triggers confirmation for at least one target reference image, at least one target reference image is identified as the object reference atlas of the reference object, and the object reference atlas of the reference object is stored in the object library.

[0134] This step enables the generated multidimensional character board to become an object asset that can be reused in subsequent storyboarding and video generation.

[0135] Step 312: Board-level referencing and target visual data generation.

[0136] When generating visual data for a reference object, a multidimensional character board can be used through board-level referencing. This board-level referencing means that the user references the multidimensional character board or a reference image related to the target shot as a reference input for the visual data generation model in storyboard or video generation.

[0137] For example, the generation constraint information in the visual data generation request can be obtained, and the generation constraint information and at least one target reference image containing the reference object can be input into the visual data generation model to obtain the target visual data. Alternatively, an associated reference image matching the generation constraint information can be determined from at least one target reference image containing the reference object, and then the generation constraint information and the associated reference image can be input into the visual data generation model to obtain the target visual data.

[0138] By inputting generation constraint information and at least one target reference image or associated reference image into the visual data generation model, the generation of target visual data is simultaneously constrained by the generation conditions defined by the generation constraint information and the appearance conditions carried by the associated reference image. This reduces the range of character appearance inferences made by the visual data generation model itself, thereby improving character consistency.

[0139] In practice, after generating at least one target reference image, a role consistency check can be performed on the target reference image to verify its consistency with the initial reference image in terms of facial features, hairstyle, clothing, accessories, and / or body proportions. If the check result meets the consistency criteria, at least one target reference image is retained; if the check result does not meet the consistency criteria, at least one step in model scoring, model determination, and / or target reference image generation is re-executed. The initial reference image can be a user-uploaded main image, a main image generated by the system based on the role definition, or a combination of a user-uploaded main image and a system-generated main image.

[0140] The image generation method provided in the embodiments of this specification breaks through the limitations of traditional three-view drawings in covering character appearance information by generating multi-dimensional character boards. After the user selects a visual generation style, this method directly generates multi-dimensional character boards covering multiple perspectives, camera tilt angles, expressions, postures, and clothing. This allows character assets to more completely describe the character's facial identity, hairstyle structure, body proportions, clothing patterns, and accessory features, providing more sufficient character references for subsequent different types of video shots, thereby solving the problem of insufficient three-view coverage.

[0141] Regarding model selection, this method selects models based on visual generation style and model capabilities, solving the problem that fixed models cannot adapt to different styles. Furthermore, it calls corresponding prompt words for the target image generation model, solving the problem that the same prompt word is not suitable for all models.

[0142] In subsequent video generation, by introducing a multi-dimensional character board, this board can be referenced in storyboarding and video generation tasks, achieving board-level referencing. The multi-dimensional character board can provide the visual data generation model with character perspectives, postures, expressions, and clothing references that are closer to the target shots, reducing the unconstrained inference of the visual data generation model on invisible character information. This provides more matching character references for different shots, improving inter-frame character consistency, identity consistency under different perspectives, cross-shot character consistency, consistency of hairstyles, clothing, and accessories, and stability of human body proportions.

[0143] Corresponding to the above method embodiments, this specification also provides embodiments of an image generation apparatus. Figure 4 A schematic diagram of an image generation apparatus according to one embodiment of this specification is shown. Figure 4 As shown, the device includes: The candidate model determination module 402 is configured to determine a set of candidate image generation models from the initial set of image generation models based on the target generation style; The target model determination module 404 is configured to score each candidate image generation model in the candidate image generation model set, obtain a scoring result, and determine a target image generation model from the candidate image generation model set based on the scoring result. Image generation module 406 is configured to input an initial reference image containing a reference object and an object generation prompt corresponding to the target image generation model into the target image generation model to generate at least one target reference image containing the reference object and the target generation style, wherein the object generation prompt contains descriptive text describing the reference object in multiple generation dimensions.

[0144] Optionally, the candidate model determination module 402 is further configured to: Obtain the model capability information of each initial image generation model in the initial image generation model set, wherein the model capability information includes the style adaptability of the initial image generation model to each generation style; Based on the target generation style and the model capability information of each initial image generation model, models whose style fit with the target generation style reaches a preset fit threshold are selected from the initial image generation model set to obtain the candidate image generation model set.

[0145] Optionally, the target model determination module 404 is further configured to: Determine at least one rating dimension corresponding to the target generation style and the dimensional weight of each rating dimension; Based on the model capability information of each candidate image generation model in the candidate image generation model set, determine the dimension score of each candidate image generation model in each scoring dimension; The scoring results of each candidate image generation model are obtained based on the dimensional scores of each candidate image generation model in each scoring dimension and the dimensional weights of each scoring dimension.

[0146] Optionally, the target model determination module 404 is further configured to: Based on the scoring results, each candidate image generation model in the candidate image generation model set is sorted in descending order to generate a sorted list, and the candidate image generation model that is first in the sorted list is determined as the target image generation model. Alternatively, the candidate image generation models in the candidate image generation model set can be sorted in descending order according to the scoring results to generate a sorted list. The sorted list can be returned to the client, and the target image generation model can be determined from the sorted list in response to the client's selection operation in the sorted list. Alternatively, the candidate image generation model in the candidate image generation model set whose scoring results exceed a preset scoring threshold can be determined as the target image generation model.

[0147] Optionally, the image generation module 406 is further configured to: The object generation prompts corresponding to the target image generation model are invoked, wherein the object generation prompts include dimension prompts for at least two of the generation dimensions: viewpoint dimension, expression dimension, posture dimension, and clothing dimension.

[0148] The device further includes: The storage module is configured to, in response to a client's confirmation operation on the at least one target reference image, determine an object reference atlas of the reference object based on the at least one target reference image, and store the object reference atlas of the reference object in an object library, wherein the object reference atlas is used to provide a reference for generating target visual data of the reference object.

[0149] The device further includes: The visual data generation module is configured to receive a visual data generation request for the reference object, obtain generation constraint information from the visual data generation request, and input the generation constraint information and at least one target reference image containing the reference object into the visual data generation model to obtain target visual data.

[0150] Optionally, the visual data generation module is further configured to: From at least one target reference image containing the reference object, determine an associated reference image that matches the generated constraint information; The generated constraint information and the associated reference image are input into the visual data generation model to obtain the target visual data.

[0151] Optionally, the visual data generation module is further configured to: The dimension constraint information of the target generation dimension in the generation constraint information is determined. From at least one target reference image containing the reference object, at least one candidate reference image of the target generation dimension is determined. And an associated reference image is determined from the at least one candidate reference image according to the dimension constraint information.

[0152] The above is an illustrative scheme of an image generation apparatus according to this embodiment. It should be noted that the technical solution of this image generation apparatus and the technical solution of the image generation method described above belong to the same concept. For details not described in detail in the technical solution of the image generation apparatus, please refer to the description of the technical solution of the image generation method described above.

[0153] Figure 5 A structural block diagram of a computing device 500 according to one embodiment of this specification is shown. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0154] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0155] In one embodiment of this specification, the above-described components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0156] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 500 can also be a mobile or stationary server.

[0157] The processor 520 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described image generation method.

[0158] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computing device embodiments are basically similar to the image generation method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the image generation method embodiments.

[0159] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the image generation method described above.

[0160] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computer-readable storage medium embodiments are relatively simple in description because they are fundamentally similar to the image generation method embodiments; relevant parts can be referred to in the description of the image generation method embodiments.

[0161] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the image generation method described above.

[0162] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the image generation method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the image generation method described above.

[0163] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0164] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0165] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0166] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0167] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An image generation method, comprising: A set of candidate image generation models is determined from the initial set of image generation models based on the target generation style; Each candidate image generation model in the candidate image generation model set is scored to obtain a score result, and a target image generation model is determined from the candidate image generation model set based on the score result. An initial reference image containing the reference object and object generation prompts corresponding to the target image generation model are input into the target image generation model to generate at least one target reference image containing the reference object and the target generation style. The object generation prompts contain descriptive text describing the reference object in multiple generation dimensions.

2. The image generation method according to claim 1, wherein determining the candidate image generation model set from the initial image generation model set according to the target generation style includes: Obtain the model capability information of each initial image generation model in the initial image generation model set, wherein the model capability information includes the style adaptability of the initial image generation model to each generation style; Based on the target generation style and the model capability information of each initial image generation model, models whose style fit with the target generation style reaches a preset fit threshold are selected from the initial image generation model set to obtain the candidate image generation model set.

3. The image generation method according to claim 1, wherein scoring each candidate image generation model in the candidate image generation model set to obtain a scoring result includes: Determine at least one rating dimension corresponding to the target generation style and the dimensional weight of each rating dimension; Based on the model capability information of each candidate image generation model in the candidate image generation model set, determine the dimension score of each candidate image generation model in each scoring dimension; The scoring results of each candidate image generation model are obtained based on the dimensional scores of each candidate image generation model in each scoring dimension and the dimensional weights of each scoring dimension.

4. The image generation method according to claim 1, wherein determining the target image generation model from the candidate image generation model set based on the scoring result includes: Based on the scoring results, each candidate image generation model in the candidate image generation model set is sorted in descending order to generate a sorted list, and the candidate image generation model that is first in the sorted list is determined as the target image generation model. Alternatively, the candidate image generation models in the candidate image generation model set can be sorted in descending order according to the scoring results to generate a sorted list. The sorted list can be returned to the client, and the target image generation model can be determined from the sorted list in response to the client's selection operation in the sorted list. Alternatively, the candidate image generation model in the candidate image generation model set whose scoring results exceed a preset scoring threshold can be determined as the target image generation model.

5. The image generation method according to claim 1, further comprising, before inputting an initial reference image containing the reference object and an object generation prompt word corresponding to the target image generation model into the target image generation model to generate at least one target reference image containing the reference object and the target generation style: The object generation prompts corresponding to the target image generation model are invoked, wherein the object generation prompts include dimension prompts for at least two of the generation dimensions: viewpoint dimension, expression dimension, posture dimension, and clothing dimension.

6. The image generation method according to claim 1, further comprising, after inputting an initial reference image containing the reference object and an object generation prompt word corresponding to the target image generation model into the target image generation model to generate at least one target reference image containing the reference object and the target generation style, the method further comprises: In response to the client's confirmation operation of the at least one target reference image, an object reference atlas of the reference object is determined based on the at least one target reference image, and the object reference atlas of the reference object is stored in an object library, wherein the object reference atlas is used to provide a reference for generating target visual data of the reference object.

7. The image generation method according to any one of claims 1-6, further comprising, after inputting an initial reference image containing the reference object and an object generation prompt word corresponding to the target image generation model into the target image generation model to generate at least one target reference image containing the reference object and the target generation style, the method further comprises: Receive a visual data generation request for the reference object, and obtain generation constraint information from the visual data generation request; The generation constraint information and at least one target reference image containing the reference object are input into the visual data generation model to obtain target visual data.

8. The image generation method according to claim 7, wherein inputting the generation constraint information and at least one target reference image containing the reference object into a visual data generation model to obtain target visual data includes: From at least one target reference image containing the reference object, determine an associated reference image that matches the generated constraint information; The generated constraint information and the associated reference image are input into the visual data generation model to obtain the target visual data.

9. The image generation method according to claim 8, wherein determining an associated reference image matching the generation constraint information from at least one target reference image containing the reference object comprises: The dimension constraint information of the target generation dimension in the generation constraint information is determined. From at least one target reference image containing the reference object, at least one candidate reference image of the target generation dimension is determined. And an associated reference image is determined from the at least one candidate reference image according to the dimension constraint information.

10. An image generation apparatus, comprising: The candidate model determination module is configured to determine a set of candidate image generation models from the initial set of image generation models based on the target generation style; The target model determination module is configured to score each candidate image generation model in the candidate image generation model set, obtain a score result, and determine a target image generation model from the candidate image generation model set based on the score result. The image generation module is configured to input an initial reference image containing a reference object and an object generation prompt corresponding to the target image generation model into the target image generation model, and generate at least one target reference image containing the reference object and the target generation style, wherein the object generation prompt contains descriptive text describing the reference object in multiple generation dimensions.

11. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the image generation method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the image generation method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the image generation method according to any one of claims 1 to 9.

Images