Image generation method, device, product, equipment and medium

Abstract

The present application discloses an image generation method, apparatus, product, device and medium, the method comprising: obtaining an object representation feature for characterizing the content semantics of a target object, obtaining N style models and M style hint information associated with the N style models, one style hint information indicating the style semantics of a style type; calling the N style models to generate a style image feature corresponding to each style hint information under the constraints of the object representation feature and the associated style hint information, one style image feature including the content feature of the target object and the style feature of the style type indicated by the corresponding style hint information; converting the style image features corresponding to the M style hint information to generate a style image of the target object under M style types. By adopting the present application, the generation efficiency of the style image of the target object under multiple style types can be improved, and the quality of the style image generated for the target object can be guaranteed.

Description

Technical Field

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an image generation method, apparatus, product, equipment and medium. Background Art

[0002] With the continuous development of computer technology, the techniques for image processing are becoming more and more abundant. For example, in some image processing scenarios, it is necessary to batch generate multiple images with the same object (such as a person) but different styles.

[0003] In existing applications, an image of the object is first generated, and then the object can be cut out from the image by manual clipping, and then the cut out object is pasted into background images of different styles to obtain multiple images of the object in different styles. This method of image generation is not only time-consuming and labor-intensive, but also objects pasted into different background images are prone to have a sense of edge, resulting in poor results in generating multiple styles of images for the object. Therefore, how to efficiently and high-quality batch generate multiple styles of images of the object is a hot issue. Summary of the invention

[0004] The present application provides an image generation method, apparatus, product, device and medium, which can improve the generation efficiency of style images of target objects under various style types and ensure the high quality of the style images generated for the target objects.

[0005] On one hand, the present application provides an image generation method, the method comprising:

[0006] Obtaining an object representation feature of a target object, where the object representation feature is used to represent the content semantics of the target object;

[0007] Obtaining N style models and M style hint information associated with the N style models, where one style hint information is used to indicate the style semantics of a style type, and one style model is associated with one or more style hint information, and both N and M are positive integers, and N is less than or equal to M;

[0008] Calling N style models to generate style image features corresponding to each style hint information under the constraints of the object representation features and the associated style hint information, wherein a style image feature includes a content feature of the target object and a style feature of the style type indicated by the corresponding style hint information, wherein the content feature is generated by performing feature learning on the content semantics represented by the object representation features, and the style feature is generated by performing feature learning on the style semantics of the style type indicated by the corresponding style hint information;

[0009] The style image features corresponding to the M style hint information are converted respectively to generate style images of the target object under M style types.

[0010] In one aspect, the present application provides an image generating device, the device comprising:

[0011] A first acquisition module is used to acquire an object representation feature of a target object, where the object representation feature is used to characterize the content semantics of the target object;

[0012] A second acquisition module is used to acquire N style models and M style prompt information associated with the N style models, where one style prompt information is used to indicate the style semantics of a style type, and one style model is associated with one or more style prompt information, and both N and M are positive integers, and N is less than or equal to M;

[0013] A feature generation module, used to call N style models to generate style image features corresponding to each style hint information under the constraints of the object representation features and the associated style hint information, wherein a style image feature includes a content feature of the target object and a style feature of the style type indicated by the corresponding style hint information, wherein the content feature is generated by performing feature learning on the content semantics represented by the object representation features, and the style feature is generated by performing feature learning on the style semantics of the style type indicated by the corresponding style hint information;

[0014] The image generation module is used to convert the style image features corresponding to the M style prompt information respectively to generate style images of the target object under M style types.

[0015] Optionally, any one of the N style models is a target style model, any style hint information associated with the target style model is the target style hint information, and the feature generation module calls the N style models to generate a style image feature corresponding to each style hint information under the constraints of the object representation feature and the associated style hint information, including:

[0016] The target style model is called to generate object constraint features of the target object under the constraints of the object representation features, and the object constraint features are used to reflect the content features of the target object;

[0017] Calling the target style model to generate style constraint features of the target object under the constraints of the target style prompt information, where the style constraint features are used to reflect the style features of the style type indicated by the target style prompt information;

[0018] The object constraint features and style constraint features are fused to generate style image features corresponding to the target style prompt information.

[0019] Optionally, the target style model includes K feature interaction sub-models connected in series for feature generation, and any feature interaction sub-model includes an object feature interaction branch and a style feature interaction branch connected in parallel, where K is a positive integer;

[0020] The target style model has a first initial noise map to be denoised, the K feature interaction sub-models include an i-th feature interaction sub-model, i is an integer, and i is less than or equal to K, and the i-th feature interaction sub-model has an input noise map;

[0021] If i is equal to 1, the input noise map is the first initial noise map. If i is greater than 1, the input noise map is the output feature of the i-1th feature interaction sub-model of the target style model. The output feature is obtained by fusing the feature generated by the object feature interaction branch in the i-1th feature interaction sub-model with the feature generated by the style feature interaction branch.

[0022] Optionally, the feature generation module calls the target style model to generate object constraint features of the target object under the constraints of the object representation features, including:

[0023] Call the object feature interaction branch in the i-th feature interaction sub-model, perform denoising on the input noise map under the constraint of the object representation feature, and generate the object constraint sub-feature of the i-th feature interaction sub-model for the target object;

[0024] Determine the object constraint sub-feature of the target object for the Mth feature interaction sub-model of the target style model as the object constraint feature;

[0025] The process of performing denoising on the input noise map under the constraint of the object representation feature is a process of interactively processing the object representation feature and the input noise map.

[0026] Optionally, the feature generation module calls the target style model to generate style constraint features of the target object under the constraints of the target style prompt information, including:

[0027] Call the style feature interaction branch in the i-th feature interaction sub-model, perform denoising on the input noise map under the constraint of the target style prompt information, and generate the initial style constraint feature of the i-th feature interaction sub-model for the target object;

[0028] Based on the initial style constraint features, generate the style constraint sub-features of the i-th feature interaction sub-model for the target object;

[0029] Determine the style constraint sub-feature of the target object in the Mth feature interaction sub-model of the target style model as the style constraint feature;

[0030] Among them, the process of denoising the input noise map under the constraint of the target style hint information is the process of interactively processing the features of the target style hint information and the input noise map.

[0031] Optionally, the style feature interaction branch of the i-th feature interaction sub-model includes a feature controller, and the feature controller is used to control the fusion strength of the features generated by the style feature interaction branch of the i-th feature interaction sub-model when they are fused with the features generated by the object feature interaction branch of the i-th feature interaction sub-model;

[0032] The feature generation module generates the style constraint sub-feature of the target object of the i-th feature interaction sub-model based on the initial style constraint feature, including:

[0033] Based on the feature controller, the initial style constraint feature is controlled and processed to generate the style constraint sub-feature of the target object of the i-th feature interaction sub-model.

[0034] Optionally, the feature controller is configured as a switch control mode or a weight control mode, the switch control mode has an on state and an off state, and the weight control mode has an associated configured fusion weight;

[0035] The feature generation module controls and processes the initial style constraint features based on the feature controller to generate the style constraint sub-features of the i-th feature interaction sub-model for the target object, including:

[0036] If the feature controller is configured as a switch control mode, and the switch control mode is in an on state, the initial style constraint feature is determined as the style constraint sub-feature of the i-th feature interaction sub-model for the target object;

[0037] If the feature controller is configured as the switch control mode and the switch control model is in the off state, it is determined that the style constraint sub-feature of the target object of the i-th feature interaction sub-model is empty;

[0038] If the feature controller is configured in weight control mode, the initial style constraint feature is weighted based on the fusion weight to generate the style constraint sub-feature of the target object of the i-th feature interaction sub-model.

[0039] Optionally, the object constraint feature includes L object channel features on L feature channels, and the style constraint feature includes L style channel features on L feature channels, where L is a positive integer;

[0040] The feature generation module fuses the object constraint features and the style constraint features to generate the style image features corresponding to the target style prompt information, including:

[0041] The object channel features and style channel features on the same feature channel are summed up respectively to generate the summed features of the target style model on each feature channel;

[0042] The object channel features and style channel features on the same feature channel are respectively concatenated in the first step to generate concatenated features of the target style model on each feature channel;

[0043] Combine the summed features and concatenated features of the target style model on L feature channels to generate style image features corresponding to the target style prompt information;

[0044] Among them, the fusion processing includes addition processing, the first step of splicing processing and combination processing.

[0045] Optionally, the feature generation module combines the summed features and concatenated features of the target style model on L feature channels to generate a style image feature corresponding to the target style prompt information, including:

[0046] The summed features and concatenated features of the target style model on the same feature channel are concatenated in the second step to generate transition fusion features of the target style model on each feature channel;

[0047] Transform the transition fusion features of the target style model on L feature channels to generate style image features corresponding to the target style prompt information;

[0048] The combined processing includes the second step of splicing processing and transformation processing.

[0049] Optionally, the feature generation module transforms the transition fusion features of the target style model on L feature channels to generate style image features corresponding to the target style prompt information, including:

[0050] Get the convolution model; the convolution model contains L convolution kernels;

[0051] L convolution kernels are used to perform convolution processing on the transition fusion features of the target style model on L feature channels respectively, to generate convolution features of the target style model on L feature channels, and one convolution kernel is used to perform convolution processing on the transition fusion features of the target style model on one feature channel;

[0052] Call the convolution model to merge the convolution features of the target style model on L feature channels to generate style image features corresponding to the target style prompt information;

[0053] Among them, the transformation processing includes convolution processing and merging processing.

[0054] Optionally, the image generation module converts the M style image features corresponding to the M style prompt information respectively to generate style images of the target object under the M style types, including:

[0055] Call the image decoder to decode each style image feature separately to generate the style image of the target object under M style types;

[0056] The style image feature corresponding to a style hint information is used to decode and generate a style image of a target object under a style type, and the conversion process includes a decoding process.

[0057] Optionally, the first acquisition module acquires the object representation feature of the target object in a manner including:

[0058] Obtaining object representation information of the target object and a second initial noise map to be denoised;

[0059] The target denoising model is called to perform denoising on the second initial noise map under the constraint of the object representation information to generate object representation features.

[0060] Optionally, the first acquisition module calls the target denoising model to perform denoising on the second initial noise image under the constraint of the object representation information to generate the object representation feature, including:

[0061] Calling the target denoising model to perform denoising on the second initial noise map under the constraint of the object representation information, and generating a predicted noise map of the target denoising model for the second initial noise map;

[0062] The second initial noise map and the predicted noise map are subjected to difference processing to generate object representation features.

[0063] Optionally, the object representation information is obtained by parsing an image generation request sent by an image client, the image generation request further carries M style prompt information, the image client includes H style prompt information that can be selected, the H style prompt information includes a full amount of style prompt information associated with a full amount of style models, the M style prompt information is selected from the H style prompt information, H is a positive integer, and H is greater than or equal to M;

[0064] The image generation module is also used to:

[0065] Return the style images of the target object under M style types to the image client;

[0066] The image client is used to associate and display the style images of the target object under M style types with the corresponding style prompt information respectively.

[0067] In one aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes a method in one aspect of the present application.

[0068] In one aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes the method in the above aspect.

[0069] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method provided in various optional modes such as the above-mentioned one aspect.

[0070] The present application can obtain the object representation feature of the target object, the object representation feature is used to characterize the content semantics of the target object; and can obtain N style models and M style hint information associated with the N style models, one style hint information is used to indicate the style semantics of a style type, one style model is associated with one or more style hint information, N and M are both positive integers, and N is less than or equal to M; thus, the N style models can be called to generate style image features corresponding to each style hint information under the constraints of the object representation feature and the associated style hint information, one style image feature includes the content feature of the target object and the style feature of the style type indicated by the corresponding style hint information, the content feature is generated by feature learning of the content semantics represented by the object representation feature, and the style feature is generated by feature learning of the style semantics of the style type indicated by the corresponding style hint information; and the M style image features corresponding to the M style hint information can be converted respectively to generate a style image of the target object under M style types. It can be seen that the method proposed in the present application guides each style model to generate style image features associated with each style hint information through the same object representation features of the target object and the style hint information associated with each style model. Each style image feature can include not only content features learned from the content semantics of the target object, but also style features learned from the style semantics of the style type indicated by the corresponding style hint information. Therefore, high-quality style images of the target object in various style types can be quickly generated through each style image feature. Each style image can contain the content of the same target object, but the content of the target object can have different style types in each style image. Therefore, the generation efficiency of the style image of the target object is improved, and the high quality of the style image of the target object is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0072] Figure 1 It is a structural diagram of a network architecture for image processing provided by an embodiment of the present application;

[0073] Figure 2 is a schematic diagram of a scene for generating a style image of a target object provided by an embodiment of the present application;

[0074] Figure 3 It is a flowchart of an image generation method provided in an embodiment of the present application;

[0075] Figure 4 is another schematic diagram of a scene for generating a style image of a target object provided by an embodiment of the present application;

[0076] Figure 5 It is a schematic diagram of a flow chart of generating style image features corresponding to style prompt information provided by an embodiment of the present application;

[0077] Figure 6 It is a structural schematic diagram of a feature interaction sub-model provided in an embodiment of the present application;

[0078] Figure 7 This is a schematic diagram of a scene for fusing object constraint features and style constraint features provided by an embodiment of the present application;

[0079] Figure 8 This is a schematic diagram of the effect of generating style images of a target object in multiple style types provided by an embodiment of the present application;

[0080] Fig. 9 It is a schematic diagram of a process of generating an object representation feature of a target object provided by an embodiment of the present application;

[0081] Fig.10 It is a schematic diagram of a scenario for generating an object representation feature of a target object provided by an embodiment of the present application;

[0082] Fig.11 is another schematic diagram of a scene for generating an object representation feature of a target object provided by an embodiment of the present application;

[0083] Fig.12 is a structural schematic diagram of an image generating device provided in an embodiment of the present application;

[0084] Fig.13 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0086] First of all, it needs to be explained that this application involves machine learning in artificial intelligence, and all data collected by this application (such as object representation features of the target object, style models, style prompt information and other related data) are collected with the consent and authorization of the object to which the data belongs (such as users, institutions or enterprises), and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of the relevant regions.

[0087] Here, the relevant technical concepts involved in this application are explained:

[0088] Latent: refers to latent variables, that is, variables that are not directly observed in the model. They are the most basic features in the entire model processing process. These variables are very important for explaining the observed data and the behavior of the model.

[0089] Cross attention: refers to the cross attention mechanism in the model, which is usually used to process the association between multiple input sequences in order to capture important information interactions between different sequences in the model. The cross attention mechanism can help the model better understand the association between different parts, so as to more accurately predict the relevant changes and impacts in the model processing process.

[0090] Self attention: refers to the self-attention mechanism in the model. The role of self attention is to further integrate all semantic representations in the latent features, while making the representation of the generated objects clearer and improving the network's noise reduction ability for latent features.

[0091] Clip Model: is an advanced deep learning model that can understand text and images simultaneously. It is a multimodal model that can match text and images, identify the content in the image and the language describing the image.

[0092] U-Net: It is an algorithm network that uses a fully convolutional network for semantic segmentation. It uses a symmetrical U-shaped structure that includes a compression path and an expansion path. The U-Net network is also the basic backbone model of the stable diffusion model. Self attention and cross attention modules are added to the U-Net network to achieve the construction of the model denoising function.

[0093] The U-Net network in this application can be divided into three parts, including Down block (downsampling module), Midblock (middle sampling module) and Up block (upsampling module). The blocks (network blocks) in these three parts are the same, but the direction of the feature size dimension is different (for example, the feature scale can be from large to small, remain unchanged, and from small to large).

[0094] Stable diffusion: is a variant of the diffusion model, called "latent diffusion model" (LDM). It can be used to eliminate continuously applied Gaussian noise in an image. Stable diffusion can be regarded as a series of denoising autoencoders. The Stable diffusion model may include a VAE (variational autoencoder) encoder, a VAE decoder, and a U-Net network. However, the Stable diffusion model is modified in this application, and the VAE encoder and VAE decoder in the Stable diffusion model are removed, while the U-Net network in the Stable diffusion model is retained.

[0095] ID: refers to the main content object (can be specified by the user, such as the target object described below) that needs to maintain consistency in appearance or attributes in the image during the same batch of image sets generation process. For example, when generating a set of image sets of multiple styles of a person, the appearance and / or clothing of the person in each image needs to be consistent, so that the IDs in the batch generated image sets can remain consistent.

[0096] See also Figure 1 , Figure 1 Schematic diagram of a network architecture for image processing provided by an embodiment of the present application. Figure 1 As shown, the network architecture may include a server 200 and a terminal device cluster, and the terminal device cluster may include one or more terminal devices, and the number of terminal devices is not limited here. Figure 1As shown, the multiple terminal devices may specifically include terminal device 1, terminal device 2, terminal device 3, ..., terminal device n; Figure 1 As shown, terminal device 1, terminal device 2, terminal device 3, ..., terminal device n can all be connected to the server 200 through a network, so that each terminal device can exchange data with the server 200 through the network connection.

[0097] like Figure 1 The server 200 shown can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (content distribution network), and big data and artificial intelligence platforms. The terminal device can be: a smart phone, a tablet computer, a laptop, a desktop computer, a smart TV, a car terminal, a smart home, and other smart terminals. The following takes the communication between the terminal device 1 and the server 200 as an example to describe the specific embodiments of the present application.

[0098] The terminal device 1 may have an image client, and the server 200 may be the background server of the image client. The image client may be used to initiate an image generation request for a target object to the server 200, so as to request the server 200 to generate style images of the target object under various style types, as described below.

[0099] Please also see Figure 2 , Figure 2 is a schematic diagram of a scene for generating a style image of a target object provided by an embodiment of the present application. Figure 2 As shown, the server 200 can obtain the object representation feature of the target object, and the object representation feature can be used to characterize the content semantics of the target object.

[0100] The server 200 can also obtain N style models (including style model m1, style model m2 and style model m3 here), and the server 200 can also obtain M style hint information associated with the N style models. One style hint information is used to indicate the style semantics of a style type (i.e., the meaning of a style type), and one style model can be associated with one or more style hint information. For example, here, the style model m1 can be associated with style hint information a1, the style model m2 can be associated with style hint information a2 and style hint information a3, and the style model m3 can be associated with style hint information a4, style hint information a5 and style hint information a6. The six style hint information can be used to indicate the six style types of the image.

[0101] Therefore, server 200 can call each style model to generate style image features corresponding to each style hint information under the constraints of the object representation features of the target object and the respectively associated style hint information. One style hint information can correspond to one style image feature, and each style image feature can include the content features of the target object, and each style image feature can also include the style features of the style type indicated by the respectively corresponding style hint information.

[0102] For example, here, the server 200 can call the style model m1 to generate the style image feature b1 corresponding to the style hint information a1 under the constraints of the style hint information a1 and the object representation feature; the server 200 can call the style model m2 to generate the style image feature b2 corresponding to the style hint information a2 under the constraints of the style hint information a2 and the object representation feature, and the server 200 can call the style model m2 to generate the style image feature b3 corresponding to the style hint information a3 under the constraints of the style hint information a3 and the object representation feature; the server 200 can call the style model m3 to generate the style image feature b4 corresponding to the style hint information a4 under the constraints of the style hint information a4 and the object representation feature, the server 200 can call the style model m3 to generate the style image feature b5 corresponding to the style hint information a5 under the constraints of the style hint information a5 and the object representation feature, and the server 200 can call the style model m3 to generate the style image feature b6 corresponding to the style hint information a6 under the constraints of the style hint information a6 and the object representation feature.

[0103] The style image t1 of the target object under the corresponding style type can be generated by the style image feature b1, the style image t2 of the target object under the corresponding style type can be generated by the style image feature b2, the style image t3 of the target object under the corresponding style type can be generated by the style image feature b3, the style image t4 of the target object under the corresponding style type can be generated by the style image feature b4, the style image t5 of the target object under the corresponding style type can be generated by the style image feature b5, and the style image t6 of the target object under the corresponding style type can be generated by the style image feature b6. The style images t1 to t6 are the style images of the six style types generated for the target object.

[0104] The server 200 may return the style images t1 to t6 to the image client in the terminal device 1 , and the image client may output and display the style images t1 to t6 for the user to view and use.

[0105] By adopting the method provided in the present application, the object representation features of the target object itself can be unified as the constraints of multiple style models to guide each style model to batch generate style images of the target object under the corresponding style type, thereby improving the generation efficiency of the style image of the target object and ensuring the consistency of the target image in each style image, thereby also improving the generation quality and effect of the style image of the target object.

[0106] See also Figure 3 , Figure 3 : is a flowchart of an image generation method provided by an embodiment of the present application. The execution subject in the embodiment of the present application may be an image generation device (hereinafter referred to as a generation device), which may be a computer device or a computer device cluster composed of multiple computer devices. The computer device may be a server, a terminal device, or other devices, without limitation. Figure 3 As shown, the method may include:

[0107] Step S101: obtaining object representation features of a target object, where the object representation features are used to characterize the content semantics of the target object.

[0108] Specifically, the target object can be any object for which image generation is required, and the present application can be used to batch generate images of the target object in various styles. For example, the target object can be any animated image, any game character, any public figure, any animal, any plant, any static object, etc.

[0109] The generating device can obtain the object representation feature of the target object, which can be obtained by learning the features of the target object, and can be used to characterize the content semantics of the target object. The object representation feature can be a latent feature, such as the object representation feature can be a latent feature generated by the target object through a related model, that is, the object representation feature can be used to potentially reflect the semantics of the object content (such as the object shape, appearance, outline, color, behavior, etc.) of the target object. In other words, the object representation feature can be used to potentially represent the target object. Optionally, the object representation feature can be a feature in the form of a vector or a feature in the form of a matrix.

[0110] Optionally, the generating device may obtain an image generating request for a target object sent by an image client, and the image generating request may be used to request generating images of various styles of the target object. The generating device may start to execute the process of generating an image for the target object after receiving the image generating request. For example, the generating device may obtain the object representation feature of the target object after receiving the image generating request.

[0111] The specific process of obtaining the object representation features of the target object can be found in the following Fig. 9 Corresponding to the relevant description in the embodiment.

[0112] Step S102 , obtaining N style models and M style hint information associated with the N style models, where one style hint information is used to indicate the style semantics of a style type, and one style model is associated with one or more style hint information, and both N and M are positive integers, and N is less than or equal to M.

[0113] Specifically, the generating device may obtain N style models and M style hint information associated with the N style models. One style hint information may be used to indicate the style semantics of a style type. In other words, one style hint information may be used to refer to (or include) the meaning of a style type. One style model among the N style models may be associated with at least one (i.e., one or more) style hint information among the M style hint information. Both N and M are positive integers (positive integers), and N may be less than or equal to M.

[0114] The M style prompt information may be carried in the image generation request sent by the above-mentioned image client, that is, the image generation request may carry the M style prompt information, and the M style prompt information may be selected or entered by the user in the image client.

[0115] Optionally, the image client may include H style prompt information that can be selected, that is, the H style prompt information is selectable by the user in the image client, H is a positive integer, and H is greater than or equal to M. The H style prompt information may include a full amount of style prompt information associated with a full amount of style models, the M style prompt information may be selected by the user from the H style prompt information in the image client, and the M style prompt information may be part or all of the H style prompt information.

[0116] Therefore, it can be understood that the above N style models are determined based on the M style prompt information selected by the user, and the N style models also include the various style models associated with the M style prompt information selected by the user. Among them, the full amount of style models can be the N style models, or the full amount of style models can also include more style models other than the N style models, and the full amount of style prompt information (that is, the above H style prompt information that can be selected) can include all the style prompt information associated with the full amount of style models.

[0117] A style model itself can be associated with one or more style hints. The number of style hints associated with a style model indicates how many types of style images the style model is trained with, which also indicates how many types of style images the style model can be used to generate. The style hints associated with different style models can be different.

[0118] The style hint information may be text information, such as the style hint information may be a text description of the style type indicated by it. For example, a style hint information may be a text description such as "ink painting", "Lego", "ancient painting", "watercolor painting", "line painting", "puppet doll", or "3D (3-dimensional) cartoon" for indicating the style type of the image.

[0119] For example, if a style model is only associated with the style hint information of "Lego", it indicates that the style model is trained with images of the style type of Lego, and the style model can be used to generate images of the style type of Lego. For another example, if a style model is associated with the style hint information of "ancient painting" and "line drawing", it indicates that the style model is trained with images of the style type of ancient painting and images of the style type of line drawing, and the style model can be used to generate images of the style types of ancient painting and line drawing. And so on.

[0120] Optionally, in another feasible implementation, the above-mentioned M style prompt information may not be obtained by the user selecting in the image client. The present application may also support the user to input keywords of the M style types that he wants to generate for the target object in the image client, and the image generation request may carry the keyword. The generation device may also match (such as text similarity matching) from the above-mentioned H style prompt information according to the keyword carried in the image generation request to obtain the above-mentioned M style prompt information corresponding to the M style types that the user wants to generate the image. If the H style prompt information does not contain the style prompt information corresponding to the style type indicated by a certain keyword input by the user, the generation device may also return to the user (such as returning to the image client) the relevant prompt information that does not support the generation of images of this style type.

[0121] Through the above process, it can be understood that the N style models can all be style models that are trained by images of the style type indicated by the associated style hint information and can be used to generate images of the corresponding style type. Optionally, a style model can be a U-Net network, and different style models can be different U-Net networks.

[0122] For example, a style model may be obtained through iterative training of sample pairs, and the sample pairs may include style hint information associated with the style model and sample images of the style type indicated by the style hint information. The style model may generate images under the guidance of the style hint information during the training process, so that the model parameters of the style model may be corrected by the difference between the image generated by the style model and the sample image, and the goal of the correction is to minimize the difference between the image generated by the style model and the sample image (e.g., to 0). For example, the difference between the image generated by the style model and the sample image may be reflected by the mean square error between the generated image and the sample image, and the larger the mean square error, the larger the difference between the generated image and the sample image, and therefore, the model parameters of the style model may be corrected by the mean square error, and the goal of the correction is to minimize the mean square error (e.g., to 0).

[0123] The above-mentioned method of training the style model is only an exemplary description. In actual application scenarios, the style model can also be obtained by training in other ways, and this application does not impose any limitation on this.

[0124] Step S103, calling N style models to generate style image features corresponding to each style hint information under the constraints of the object representation features and the associated style hint information, wherein a style image feature includes a content feature of the target object and a style feature of the style type indicated by the corresponding style hint information, wherein the content feature is generated by performing feature learning on the content semantics represented by the object representation features, and the style feature is generated by performing feature learning on the style semantics of the style type indicated by the corresponding style hint information.

[0125] Specifically, the generation model can call the above N style models, and generate style image features corresponding to each style hint information under the constraints of the object representation features of the target object and the style hint information associated with each style model. One style hint information can correspond to one style image feature, and the style image feature corresponding to one style hint information can be used to generate an image of the style type indicated by the style hint information (which can be called a style image) for the target object.

[0126] A style image feature may include the content feature of the target object and the style feature of the style type indicated by the corresponding style hint information. The content feature may be obtained by the corresponding style model under the constraint of the object representation information, by learning the content semantics represented (or contained) by the object representation information (such as by learning through a cross-attention mechanism). Similarly, the style feature may also be obtained by the corresponding style model under the constraint of the corresponding style hint information, by learning the style semantics of the style type indicated by the style hint information (such as by learning through a cross-attention mechanism). In other words, each style image feature will include the content feature of the same target object, and each style image feature will also include the style feature of the style type indicated by its corresponding style hint information.

[0127] Therefore, each style image feature can be used to generate images with the same target object (which can be called ID consistency, and the ID is the object that needs to be kept consistent) but different style types.

[0128] The specific process of how each style model generates the style image features corresponding to each style hint information under the constraints of the object representation features and the associated style hint information can be seen below. Figure 5 Corresponding to the relevant description in the embodiment.

[0129] Step S104 , converting the style image features corresponding to the M style prompt information respectively to generate style images of the target object under M style types.

[0130] Specifically, the generating device may convert the style image features (there may be M style image features) corresponding to the above-mentioned M style hint information respectively to generate images of the target object under M style types (which may be called style images). The style image feature corresponding to one style hint information may be used to convert and generate a style image of the target object under one style type.

[0131] For example, the generating device may obtain an image decoder, which is also a decoder obtained through training (which may be jointly trained with each style model) and can be used to decode the features generated by the style model to generate a corresponding image. For example, the image decoder may be a VAE decoder. Each style model may share the same image decoder.

[0132] Therefore, the generating device may call the image decoder to perform decoding processing on each style image feature respectively to generate a style image of the target object under M style types. A style image feature may be used to decode and generate a style image of the target object under one style type. The conversion processing of the style image feature may include the decoding processing of the style image feature.

[0133] See also Figure 4 , Figure 4 FIG. 1 is another schematic diagram of a scene for generating a style image of a target object provided by an embodiment of the present application. Figure 4 As shown, the above-mentioned M style prompt information may include style prompt information 1 to style prompt information M here, and each style prompt information corresponds to its own style image feature, such as style prompt information 1 corresponds to style image feature 1 ~ style prompt information M corresponds to style image feature M.

[0134] Therefore, the generating device can generate style images of the target object under various style types through various style image features. For example, style image feature 1 can be used to generate style image 1 of the target object under the style type indicated by style prompt information 1, and style image feature 2 can be used to generate style image 2 of the target object under the style type indicated by style prompt information 1, and so on.

[0135] Through the above process, style images of the target object under M style types (or M) are generated. For example, the target object may be a giant panda, and the M style types may include a style type of ink painting, a style type of line drawing, and a style type of Lego. Then, the generated M style images may include a giant panda of the ink painting style type, a giant panda of the line drawing style type, and a giant panda of the Lego style type.

[0136] Furthermore, the generating device may also return the style images generated for the target object under the M style types to the image client, and the image client may display the style images of the target object under the style type in the client interface for the user to view. For example, the image client may associate the style images of the target object under the M style types with the corresponding style prompt information for display.

[0137] Optionally, in another feasible implementation, the above process of generating style images of the target object under multiple style types may also be performed by the image client itself, and the image client may store or obtain from the background the relevant models and data required for the process.

[0138] The present application provides an automated method for batch generation of sets of images with consistent IDs and multiple styles, which can batch generate multiple multi-style images that maintain the consistency of the ID content at one time based on the desired main generated content ID. In this method, an innovative method for transmitting controllable information of ID content based on ID latent features (i.e., object representation features) is proposed. In this method, the ID latent features generated by a model (such as the target denoising model described below) are used as the transmission information, and a controllable fusion method of multiple style models is combined, so that high-quality generation representations with consistent ID content can be maintained in multiple style models, and it can also be fused with multiple style models, so as to batch generate style sets of finished products with diverse styles (i.e., style images of the target object under the above-mentioned M style types).

[0139] The present application can obtain the object representation feature of the target object, the object representation feature is used to characterize the content semantics of the target object; and can obtain N style models and M style hint information associated with the N style models, one style hint information is used to indicate the style semantics of a style type, one style model is associated with one or more style hint information, N and M are both positive integers, and N is less than or equal to M; thus, the N style models can be called to generate style image features corresponding to each style hint information under the constraints of the object representation feature and the associated style hint information, one style image feature includes the content feature of the target object and the style feature of the style type indicated by the corresponding style hint information, the content feature is generated by feature learning of the content semantics represented by the object representation feature, and the style feature is generated by feature learning of the style semantics of the style type indicated by the corresponding style hint information; and the M style image features corresponding to the M style hint information can be converted respectively to generate a style image of the target object under M style types. It can be seen that the method proposed in the present application guides each style model to generate style image features associated with each style hint information through the same object representation features of the target object and the style hint information associated with each style model. Each style image feature can include not only content features learned from the content semantics of the target object, but also style features learned from the style semantics of the style type indicated by the corresponding style hint information. Therefore, high-quality style images of the target object in various style types can be quickly generated through each style image feature. Each style image can contain the content of the same target object, but the content of the target object can have different style types in each style image. Therefore, the generation efficiency of the style image of the target object is improved, and the high quality of the style image of the target object is guaranteed.

[0140] See also Figure 5 , Figure 5FIG. 1 is a flow chart of generating style image features corresponding to style prompt information provided by an embodiment of the present application. Figure 5 As shown, the process may include:

[0141] Step S201 : calling the target style model to generate object constraint features of the target object under the constraints of the object representation features, where the object constraint features are used to reflect the content features of the target object.

[0142] Specifically, any one of the N style models mentioned above can be called a target style model, and any style hint information associated with the target style model can be called target style hint information. Since the principle of calling each style model to generate style image features corresponding to each associated style hint information is the same, the following takes calling the target style model to generate style image features corresponding to the associated target style hint information as an example for specific explanation.

[0143] Since the model structures of various style models may be the same, except that different style models may be obtained by training images of different style types and may be used to generate images of different style types, this application still uses the model structure of the target style model as an example for specific description.

[0144] The target style model may include K feature interaction sub-models for feature generation. The K feature interaction sub-models may be connected in series in the target style model. K is a positive integer. The model structure of each feature interaction sub-model may also be the same.

[0145] For example, the target style model can be a U-Net network, which can contain multiple U-Netblocks. A U-Net block can be understood as a basic network unit in the U-Net network. A U-Netblock can be a feature interaction sub-model. A U-Net block can include three modules, which can include a Resnet module (residual module), a self-attention module (self-attention mechanism module) and a cross attention module (cross attention mechanism module). The specific network structure of the U-Net block can also be seen below. Figure 6 Corresponding to the relevant description in the embodiment.

[0146] Any feature interaction sub-model may include a parallel object feature interaction branch and a style feature interaction branch, that is, each feature interaction sub-model may include two parallel network branches, the object feature interaction branch and the style feature interaction branch. The object feature interaction branch may be a branch for generating features related to the object content of the target object, and the style feature interaction branch may be a branch for generating features related to the corresponding style type.

[0147] The target style model may have a first initial noise map to be denoised, and the first initial noise map may be a feature map of randomly generated Gaussian noise. The first initial noise map may be the input of the entire target style model, and the first initial noise map may also be latent (which may be referred to as initial noise latent).

[0148] The above-mentioned K feature interaction sub-models may include the i-th feature interaction sub-model, where i is a positive integer and i is less than or equal to K, that is, the i-th feature interaction sub-model may be any one of the K feature interaction sub-models. Since the process of feature generation for each feature interaction sub-model is the same, the process of feature generation through the i-th feature interaction sub-model is taken as an example for specific explanation here.

[0149] The i-th feature interaction sub-model may have an input noise map, which is the input of the i-th feature interaction sub-model. Optionally, if i is equal to 1, that is, the i-th feature interaction sub-model is the first feature interaction sub-model in the target style model, then the input noise map may be the first initial noise map mentioned above, and if i is greater than 1, that is, the i-th feature interaction sub-model is not the first feature interaction sub-model in the target style model, then the input noise map may be the output feature of the i-1th feature interaction sub-model of the target style model (that is, the output of the i-1th feature interaction sub-model), and the output feature is obtained by fusing the feature generated by the object feature interaction branch in the i-1th feature interaction sub-model with the feature generated by the style feature interaction branch. In other words, the input of the latter feature interaction sub-model in the target style model may be the output of the former feature interaction sub-model.

[0150] Therefore, the generating device may call the target style model to generate the object constraint feature of the target object under the constraint of the object representation feature. The process may include:

[0151] In fact, from the perspective of model functions, each style model (U-Net network) can also be a denoising model, which can perform denoising on the latent of the input noise under the constraints / guidance of the input conditions to generate corresponding features. Therefore, the generating device can call the object feature interaction branch in the above-mentioned i-th feature interaction sub-model, and perform denoising on the above-mentioned input noise map (i.e., the input of the i-th feature interaction sub-model) under the constraints of the object representation features of the target object, and can generate the object constraint sub-feature of the i-th feature interaction sub-model for the target object. The denoising process is to remove the noise in the input noise map that does not match the object representation features under the influence and guidance of the object representation features. Here, the features obtained after the noise removal of the input noise map can also be used as the object constraint sub-feature of the i-th feature interaction sub-model for the target object.

[0152] Among them, the process of denoising the above-mentioned input noise map under the constraint of the object representation feature of the target object can be understood as the process of shaping the features to be generated under the constraint of the object representation feature of the target object. Through this denoising process, the object feature interaction branch in the i-th feature interaction sub-model can learn the features of the object content of the target object through the input conditions (i.e., the object representation feature of the target object). The object feature interaction branch in the i-th feature interaction sub-model can be based on the cross attention module to the object representation feature of the target object, and denoise the above-mentioned input noise map.

[0153] The generative model can, according to the above principle, call the object feature interaction branch in each feature interaction sub-model in turn through the input conditions to generate the object constraint sub-feature of each feature interaction sub-model for the target object. Thus, the generative model can use the object constraint sub-feature of the target object in the Mth feature interaction sub-model (i.e., the last feature interaction sub-model) in the target style model as the object constraint feature finally generated by the target style model for the target object, and the object constraint feature can be used to characterize the content semantic features of the target object.

[0154] Among them, the process of the object feature interaction branch in the i-th feature interaction sub-model performing denoising on the input noise map under the guidance of the object representation feature can also be understood as the process of the object feature interaction branch performing feature interaction processing on the object representation feature and the input noise map.

[0155] In addition, it is understandable that the dimension of the object representation feature may not match the dimension of the feature that can be processed by the cross attention module in the object feature interaction branch. Therefore, the present application can also transform the size dimension of the object representation feature to obtain a feature that matches the dimension of the feature that can be processed by the cross attention module, so as to guide the cross attention module in the object feature interaction branch to generate object constraint sub-features for the target object.

[0156] Step S202 : calling the target style model to generate style constraint features of the target object under the constraints of the target style prompt information, where the style constraint features are used to reflect the style features of the style type indicated by the target style prompt information.

[0157] Specifically, the generating device can call the style feature interaction branch in the above-mentioned i-th feature interaction sub-model, and under the constraint of the target style prompt information, perform denoising processing on the above-mentioned input noise map (i.e., the input of the i-th feature interaction sub-model), so as to generate the initial style constraint features of the i-th feature interaction sub-model for the target object.

[0158] Optionally, the target style hint information may be text information, and the generating device may first call a trained text encoder (which may be a partial structure in the Clip model for text encoding) to perform feature encoding processing on the target style hint information to generate text encoding features of the target style hint information, i.e., embedding features of the target style hint information. Thus, the generating device may use the text encoding features as the input conditions of the style feature interaction branch in the i-th feature interaction sub-model to call the style feature interaction branch in the i-th feature interaction sub-model, and perform denoising processing on the above-mentioned input noise map under the constraints of the text encoding features of the target style hint information to generate the initial style constraint features of the i-th feature interaction sub-model for the target object.

[0159] The denoising process is that the style feature interaction branch in the i-th feature interaction sub-model removes the noise in the input noise map that is inconsistent with the text encoding feature under the influence and guidance of the text encoding feature. The feature obtained after noise removal of the input noise map can be used as the initial style constraint feature of the i-th feature interaction sub-model for the target object.

[0160] Among them, the process of denoising the above-mentioned input noise image under the constraint of the text encoding features of the target style prompt information can be understood as the process of shaping the features to be generated under the constraint of the text encoding features of the target style prompt information. Through this denoising process, the style feature interaction branch in the i-th feature interaction sub-model can learn the style features of the style type indicated by the target style prompt information through the input conditions (i.e., the text encoding features of the target style prompt information). The style feature interaction branch in the i-th feature interaction sub-model can be denoising the above-mentioned input noise image under the cross-attention of the text encoding features of the target style prompt information.

[0161] The generating device can generate the style constraint sub-feature of the target object of the i-th feature interaction sub-model through the above initial style constraint feature. For example, the generating device can control and adjust (can be referred to as regulation) the influence strength of the initial style constraint feature in the subsequent process of image generation to obtain the style constraint sub-feature of the target object of the i-th feature interaction sub-model.

[0162] In a feasible implementation, the style feature interaction branch of the i-th feature interaction sub-model may also include a feature controller, which may be used to control the fusion strength of the features generated by the style feature interaction branch of the i-th feature interaction sub-model when they are fused with the features generated by the object feature interaction branch of the i-th feature interaction sub-model. In other words, the feature controller is used to control the fusion strength of the features generated by the style feature interaction branch of the i-th feature interaction sub-model. The fusion strength may be understood as the influence of the features generated by the style feature interaction branch of the i-th feature interaction sub-model on image generation. The greater the fusion strength, the greater the influence. Conversely, the smaller the fusion strength, the smaller the influence. A feature controller may be configured in the style feature interaction branch of each feature interaction sub-model.

[0163] Therefore, the generating device can control and process the initial style constraint features generated by the style feature interaction branch of the i-th feature interaction sub-model through the feature controller in the style feature interaction branch of the i-th feature interaction sub-model to generate the style constraint sub-features of the i-th feature interaction sub-model for the target object, as described below.

[0164] The feature controller may have two control modes, including a switch control mode and a weight control mode, and the feature controller may be configured as the switch control mode or the weight control mode. The switch control mode has an on state and an off state, and the weight control mode has an associated configured fusion weight.

[0165] If the feature controller in the style feature interaction branch of the i-th feature interaction sub-model is configured as a switch control mode, and the switch control mode is in the on state, the generating device can directly use the above-mentioned initial style constraint feature as the style constraint sub-feature of the i-th feature interaction sub-model for the target object. Among them, the processing of directly using the initial style constraint feature as the style constraint sub-feature of the i-th feature interaction sub-model for the target object belongs to the above-mentioned control processing.

[0166] If the feature controller in the style feature interaction branch of the i-th feature interaction sub-model is configured as a switch control mode, and the switch control mode is in the off state, the generating device can determine that the style constraint sub-feature of the i-th feature interaction sub-model for the target object is empty (which can be understood as a 0 feature), that is, in this case, it can be understood that the style feature interaction branch in the i-th feature interaction sub-model is not used for subsequent image generation, and in the i-th feature interaction sub-model, only the features generated by the object feature interaction branch need to be used. Among them, the process of determining that the style constraint sub-feature of the i-th feature interaction sub-model for the target object is empty belongs to the above-mentioned control process.

[0167] Among them, it can be understood that when the switch control mode is in the on state, the fusion strength of the features generated by the style feature interaction branch of the i-th feature interaction sub-model must be higher than the fusion strength of the features generated by the style feature interaction branch of the i-th feature interaction sub-model when the switch control mode is in the off state.

[0168] If the feature controller in the style feature interaction branch of the i-th feature interaction sub-model is configured as a weight control mode, the generating device can perform weighted processing on the above-mentioned initial style constraint feature through the fusion weight configured in association with the weight control mode (which belongs to the above-mentioned control processing, such as multiplying the fusion weight by the initial style constraint feature) to generate the style constraint sub-feature of the i-th feature interaction sub-model for the target object.

[0169] Optionally, the feature controllers in the style feature interaction branches of different feature interaction sub-models can be configured to have different or the same control modes, and the fusion weights associated with the weight control modes of the feature controllers in the style feature interaction branches of different feature interaction sub-models, or the states of the switch control modes can be set accordingly according to the actual application scenarios.

[0170] The generative model can, according to the above principle, sequentially call the style feature interaction branch in each feature interaction sub-model through input conditions to generate the style constraint sub-feature of each feature interaction sub-model for the target object. Thus, the generative model can use the style constraint sub-feature of the target object in the Mth feature interaction sub-model (i.e., the last feature interaction sub-model) in the target style model as the style constraint feature finally generated by the target style model for the target object. The style constraint feature can be used to reflect (or include) the style feature of the style type indicated by the target style prompt information, or the style constraint feature can be directly understood as the style feature of the style type indicated by the target style prompt information.

[0171] Among them, the process of denoising the input noise map by the style feature interaction branch in the i-th feature interaction sub-model under the guidance of the text encoding features of the target style prompt information can also be understood as the process of the object feature interaction branch performing feature interaction processing on the text encoding features and the input noise map, that is, the process of performing feature interaction processing on the target style prompt information and the input noise map.

[0172] Compared with other solutions that usually control object-related features, there is a special consideration for setting the feature controller in the style feature interaction branch in this application, because the consistency of ID (i.e., target object) needs to be maintained in multiple style models, and the ID latent (i.e., the object representation feature of the target object) received in multiple style models in this solution is the same, which is the same latent, so in the process of ID latent characterization, its strength is consistent, so in order to keep the representation strength of the ID latent in each style model consistent and the quality of the ID content is not destroyed, the object feature interaction branch can be set without setting the style feature interaction branch in this application. In addition, since the training degree of different style models in this application may also be different, the strength of the representation ability of the style semantics of different style models will also be different. Therefore, in order to prevent the style semantics in the style model from excessively affecting the accurate retention of the ID content, this application is modified and added to the cross attention module of the style feature interaction branch. A switch control module (i.e., feature controller) is added to control the representation of style semantics through the switch control module, so as to achieve the purpose of not affecting the consistency of ID in the images generated by each style model.

[0173] By setting a feature controller in the style feature interaction branch, the present application can control the ratio of the features generated by the style feature interaction branch and the features generated by the object feature interaction branch when they are fused, thereby achieving control over the stylization intensity of the style image, thereby enabling more diverse sets of images to be generated for the target object.

[0174] Of course, in actual application scenarios, if there is a special need, another feature controller can be added to the object feature interaction branch to enhance or adjust the representation strength of the content semantics of the ID.

[0175] See also Figure 6 , Figure 6 It is a structural diagram of a feature interaction sub-model provided in an embodiment of the present application. Figure 6 The feature interaction submodel shown may be the i-th feature interaction submodel mentioned above (it may be any feature interaction submodel in the target style model), and the feature interaction submodel may include a residual module (which may be used to extract latent features of higher dimensions), a self-attention mechanism module (which may be used to make the semantic concepts of each element in the latent features clearer), and two parallel cross-attention mechanism modules (which may be used to support latent generation under the guidance of input conditions). Of the two parallel cross-attention mechanism modules, the cross-attention mechanism module on the left side of the figure is used to perform cross-attention processing on style hint information, and the cross-attention mechanism module on the right side of the figure is used to perform cross-attention processing on the object representation features of the target object.

[0176] Among them, the style feature interaction branch in the feature interaction submodel includes the cross attention mechanism module on the left side of the figure, and the style feature interaction branch may also include a control switch, which is the above-mentioned feature controller. The object feature interaction branch in the feature interaction submodel includes the cross attention mechanism module on the right side of the figure, and the object feature interaction branch may not include a control switch.

[0177] like Figure 6 As shown, the style latent here can be the feature finally generated by the style feature interaction branch, and the feature can be the style constraint sub-feature of the feature interaction sub-model for the target object; and the object latent here can be the object constraint sub-feature of the feature interaction sub-model for the target object.

[0178] Therefore, by fusing the style latent and the object latent, the features of the final output of the feature interaction sub-model can be generated, that is, the fused latent here. In other words, the fused latent here is the output feature of the current feature interaction sub-model.

[0179] The principle of fusing the style latent and the object latent is the same as the principle of fusing the object constraint feature and the style constraint feature described below. For details, please refer to the relevant description in the following step S203. It can be understood that the concepts of the style latent and the style constraint feature are similar, and the concepts of the object latent and the object constraint feature are also similar, except that when i is not equal to M, the style latent and the object latent are generated by different feature interaction sub-models in the target style model compared to the style constraint feature and the object constraint feature.

[0180] This application innovatively expands the feature interaction sub-model in the style model by two parallel crossattention branches. In addition to the style hint information associated with the style model being used as the input condition of a crossattention branch of the style model for corresponding calculations, the object representation features of the target object can also be used as the input condition of another parallel cross attention branch of the style model for corresponding calculations. This parallel calculation method can prevent the style model from interfering with each other when performing relevant calculations of style semantic features and relevant calculations of object content semantic features, thereby improving the ID retention of the subsequently generated style image.

[0181] At the same time, in this parallel calculation method, the object representation features of the target object can serve as a dominant guider information (guiding information, also called constraint information) of the style model, thereby increasing the representation of the ID content in the final style image.

[0182] Step S203: fusing the object constraint features and the style constraint features to generate style image features corresponding to the target style prompt information.

[0183] Specifically, the object constraint feature may include features on L feature channels (such as the three color channels of R, G, and B). The features of the object constraint feature on the L feature channels may be referred to as object channel features. One feature channel may have one object channel feature, and there are L object channel features in total. One object channel feature may be a feature map, such as a feature map of 128×128 dimensions. L is a positive integer. Among them, the feature channel may be referred to as a channel.

[0184] Similarly, the style constraint feature may also include features on the L feature channels. The features of the style constraint feature on the L feature channels may be called style channel features. One feature channel may have one style channel feature, and there are L style channel features in total. One style channel feature may also be a feature map, such as a feature map of 128×128 dimensions.

[0185] The dimension of the object channel feature and the dimension of the style channel feature may be the same, so the generating device may perform sum processing on the object channel feature and the style channel feature on the same feature channel to generate a sum feature of the target style model on each feature channel, and one feature channel may have one sum feature. The sum processing of the object channel feature and the style channel feature may refer to respectively adding the elements at the same position (such as the same number of rows and columns) in the object channel feature and the style channel feature.

[0186] In addition, the generating device can also perform concatenation processing on the object channel features and style channel features on the same feature channel (which can be called the first step of concatenation processing, i.e., concat processing) to generate concatenation features of the target style model on each feature channel. There can be one concatenation feature on one feature channel. A concatenation feature can be obtained by concatenating an object channel feature and a style channel feature on the same feature channel (such as left-right concatenation or top-bottom concatenation).

[0187] Thus, the generating device can further combine the summed features and concatenated features of the target style model on the L feature channels to finally generate the style image features corresponding to the target style prompt information, as described below. The above fusion processing of the object constraint features and the style constraint features can include the above summing processing, the first step concatenation processing and the combining processing.

[0188] Among them, the present application can maximize the retention and combination of the features of elements at each position in the object channel features and the style channel features by performing addition and splicing processing on the object channel features and the style channel features in parallel, thereby improving the accuracy of the style image features subsequently generated for the target style prompt information.

[0189] The generating device can perform splicing processing on the summed features and the spliced ​​features of the target style model on the same feature channel (which can be called the second step of splicing processing) to generate transition fusion features of the target style model on each feature channel. That is, the feature obtained by splicing the summed features and the spliced ​​features on the same feature channel (such as splicing left and right or splicing up and down according to the matching dimensions between the summed features and the spliced ​​features) is called a transition fusion feature, and one feature channel can have one transition fusion feature.

[0190] Furthermore, the generating device may transform the L transition fusion features of the target style model on the L feature channels to finally generate the style image features corresponding to the target style prompt information, as described below. The above-mentioned combined processing of the sum feature and the splicing feature may include the above-mentioned second step splicing processing and the transformation processing.

[0191] In a feasible implementation, the generating device may perform convolution processing on the above-mentioned L transition fusion features to merge the L transition fusion features into one feature, and the merged feature is the style image feature corresponding to the target style prompt information, as described below.

[0192] For example, the generating device can obtain a convolution model (also referred to as a convolution network), which can have L convolution kernels, which can be different convolution kernels, and the L convolution kernels can be one-to-one corresponding to the L transition fusion features, that is, one convolution kernel corresponds to a transition fusion feature on a feature channel, and one convolution kernel is used to perform convolution processing on the transition fusion feature on the corresponding feature channel.

[0193] Therefore, the generating device can use the L convolution kernels to perform convolution processing on the transition fusion features of the target style model on the L feature channels respectively to generate the convolution features of the target style model on the L feature channels. There can be a total of L convolution features, and one convolution feature is obtained by convolution processing a corresponding transition fusion feature with one convolution kernel.

[0194] In the present application, by using different convolution kernels to perform convolution processing on the transition fusion features on different feature channels, the features of the elements at the same position in each transition fusion feature can be more fully calculated and fused, so that the accuracy and effect of the style image features generated by the target style prompt information can be further improved in the future.

[0195] The generating device can call the convolution model to merge the L convolution features to generate the style image features corresponding to the target style prompt information. Among them, the convolution model itself can have the ability to merge multiple features (such as L convolution features). For example, the convolution model can merge multiple features into one feature through multi-scale feature fusion and other technologies. The specific method can be determined based on the structure and function of the convolution model actually obtained.

[0196] Among them, the transformation processing of the transition fusion feature can include the above-mentioned merging processing. Through the above process, the target style model is called to generate the style image features corresponding to the associated target style prompt information. The generating device can generate the style image features corresponding to the M style prompt information associated with the above N style models according to the same principle, and one style prompt information corresponds to one style image feature.

[0197] See also Figure 7 , Figure 7 Schematic diagram of a scene for fusing object constraint features and style constraint features provided by an embodiment of the present application. Figure 7 As shown, the above-mentioned L feature channels may include the three feature channels of feature channel 1, feature channel 2 and feature channel 3 herein, the object constraint feature may include object channel feature 1 on the feature channel 1, object channel feature 2 on the feature channel 2 and object channel feature 3 on the feature channel 3; and the style constraint feature may include style channel feature 1 on the feature channel 1, style channel feature 2 on the feature channel 2 and style channel feature 3 on the feature channel 3.

[0198] Therefore, the generating device can perform the sum processing on the object channel feature 1 and the style channel feature 1 to obtain the summed feature 1 on the feature channel 1, and the generating device can perform the first step of splicing processing on the object channel feature 1 and the style channel feature 1 to obtain the spliced ​​feature 1 on the feature channel 1; and, the generating device can perform the sum processing on the object channel feature 2 and the style channel feature 2 to obtain the summed feature 2 on the feature channel 2, and the generating device can perform the first step of splicing processing on the object channel feature 2 and the style channel feature 2 to obtain the spliced ​​feature 2 on the feature channel 2; and, the generating device can perform the sum processing on the object channel feature 3 and the style channel feature 3 to obtain the summed feature 3 on the feature channel 3, and the generating device can perform the first step of splicing processing on the object channel feature 3 and the style channel feature 3 to obtain the spliced ​​feature 3 on the feature channel 3.

[0199] Thus, the generating device can perform a second step of splicing processing on the summed feature 1 and the spliced ​​feature 1 on feature channel 1, and obtain the transition fusion feature 1 on feature channel 1; the generating device can perform a second step of splicing processing on the summed feature 2 and the spliced ​​feature 2 on feature channel 2, and obtain the transition fusion feature 2 on feature channel 2; and the generating device can perform a second step of splicing processing on the summed feature 3 and the spliced ​​feature 3 on feature channel 3, and obtain the transition fusion feature 3 on feature channel 3. Furthermore, the generating device can perform convolution addition and merging processing on the transition fusion feature 1, the transition fusion feature 2, and the transition fusion feature 3, and finally generate the style image feature corresponding to the target style prompt information.

[0200] See also Figure 8 , Figure 8 Schematic diagram of the effect of generating style images of a target object in multiple style types provided by an embodiment of the present application. Figure 8 As shown, the target object may be a giant panda, and the generating device may perform feature learning on the giant panda to generate object representation features of the target object.

[0201] The N style models may include style model 1 to style model 4, and the M style hint information may include style hint information 1 to style hint information 4. In the M style hint information, style model 1 may be associated with style hint information 1, style model 2 may be associated with style hint information 2, style model 3 may be associated with style hint information 3, and style model 4 may be associated with style hint information 4.

[0202] For example, the style type indicated by the style prompt information 1 may be a doll type, the style type indicated by the style prompt information 2 may be an ink painting type, the style type indicated by the style prompt information 3 may be a Lego type, and the style type indicated by the style prompt information 4 may be a line type.

[0203] Therefore, style model 1 can generate style constraint feature 1 under the constraint and guidance of object representation feature, and can generate object constraint feature 1 under the constraint and guidance of style hint information 1, and the style constraint feature 1 and the object constraint feature 1 are fused to generate style image feature 1 corresponding to style hint information 1. Similarly, style model 2 can generate style constraint feature 2 under the constraint and guidance of object representation feature, and can generate object constraint feature 2 under the constraint and guidance of style hint information 2, and the style constraint feature 2 and the object constraint feature 2 are fused to generate style image feature 2 corresponding to style hint information 2.

[0204] Similarly, style model 3 can generate style constraint feature 3 under the constraint and guidance of object representation feature, and can generate object constraint feature 3 under the constraint and guidance of style hint information 3, and the style constraint feature 3 and the object constraint feature 3 are fused to generate style image feature 3 corresponding to style hint information 3. Also, style model 4 can generate style constraint feature 4 under the constraint and guidance of object representation feature, and can generate object constraint feature 4 under the constraint and guidance of style hint information 4, and the style constraint feature 4 and the object constraint feature 4 are fused to generate style image feature 4 corresponding to style hint information 4.

[0205] The generating device decodes and processes the above-mentioned style image feature 1 to generate a style image of the giant panda in the doll type; the generating device decodes and processes the above-mentioned style image feature 2 to generate a style image of the giant panda in the ink painting type; the generating device decodes and processes the above-mentioned style image feature 3 to generate a style image of the giant panda in the Lego type; the generating device decodes and processes the above-mentioned style image feature 4 to generate a style image of the giant panda in the line type.

[0206] Through the above process of the present application, each style model is guided by its associated style hint information and object representation features, and accurately generates style image features corresponding to each style hint information. By using the style image features, high-quality style images of the target object in various style types can be quickly generated in batches.

[0207] See also Fig. 9 , Fig. 9 Schematic diagram of a process for generating an object representation feature of a target object provided by an embodiment of the present application. Fig. 9 As shown, the process may include:

[0208] Step S301: Obtain object representation information of a target object and a second initial noise map to be denoised.

[0209] Specifically, the generating device may obtain the object representation information of the target object. As the name implies, the object representation information may be information used to represent (or describe) the target object, such as the name of the target object (which is text information). The object representation information may be provided by the user, and the object representation information may be carried in the image generation request sent by the above-mentioned image client, that is, the image generation request may carry the object representation information. Therefore, the object representation information may be obtained by the generating device parsing the image generation request.

[0210] Among them, the user can provide and enter the object representation information of the corresponding target object to the image client according to his own needs. For example, if the user wants to generate style images of giant pandas in various styles, the object representation information can be the three words "giant panda"; if the user wants to generate style images of a certain animated character (such as SpongeBob SquarePants) in various styles, the object representation information can be the four words "SpongeBob SquarePants"; and so on.

[0211] The generating device can also obtain a second initial noise map to be denoised by the target denoising model, and the second initial noise map can also be a feature map of randomly generated Gaussian noise (belonging to latent), and the second initial noise map can be the input of the target denoising model. The target denoising model can be a trained Stable diffusion model, but when the Stable diffusion model is used in this application, the VAE encoder and VAE decoder in the Stable diffusion model are removed, and the U-Net network is retained.

[0212] Therefore, it can be understood that the target denoising model can be a trained model that can be used to denoise the latent of the input noise according to the input conditions (such as conditions related to the input text) to generate features that are consistent with (i.e., match) the input conditions. The training principle of the target denoising model can be the same as the training principle of the above-mentioned style model.

[0213] Among them, if the object representation information is text information, then correspondingly, the target denoising model can also be obtained by training the object-related text, so that the target denoising model can learn and recognize the relevant features of the object-related text. For example, the target denoising model can be obtained by training a sample pair constructed by a sample text and a sample image that matches the sample text (such as the image content of the sample image can be the text content described by the sample text). Optionally, the sample text can be a text used to represent (or describe) a training object (which can be any object that needs to be trained, such as a person, an animal, a plant, a stationary object, etc.), and the sample image can contain the training object represented (described) by the sample text.

[0214] Step S302: Call the target denoising model to perform denoising on the second initial noise image under the constraint of the object representation information to generate object representation features.

[0215] Specifically, the generating device may call the target denoising model to perform denoising on the second initial noise map under the constraints of the encoding features of the above-mentioned object representation information to generate object representation features of the target object, wherein the encoding features may be obtained by feature encoding the object representation information using a corresponding encoder, as described below.

[0216] The object representation information can be text information. The generating device can also call the above-mentioned trained text encoder to perform feature encoding processing on the object representation information to generate encoding features of the object representation information. The encoding features can be called information encoding features of the target object. For example, the information encoding features can be features in vector form.

[0217] Therefore, the generating device can input the information encoding feature into the target denoising model, and the target denoising model can perform denoising on the second initial noise map under the constraint of the information encoding feature of the object representation information to generate the object representation feature of the target object.

[0218] Similarly, the denoising process is to predict the noise in the second initial noise map that is inconsistent with the information coding features of the object representation information under the influence and guidance of the information coding features of the object representation information. The process of denoising the second initial noise map under the constraints of the information coding features of the object representation information can be understood as the process of shaping the features to be generated under the constraints of the information coding features of the object representation information. Through the denoising process, the target denoising model can learn the features related to the semantics of the object content of the target object through the input conditions (i.e., the information coding features of the object representation information). The target denoising model can be a denoising process for the second initial noise map under the cross-attention of the information coding features of the object representation information.

[0219] Among them, the generating device calls the target denoising model to perform denoising on the second initial noise map under the constraint of the object representation information (that is, under the constraint of the information coding characteristics of the object representation information), and can first generate a predicted noise map of the target denoising model for the second initial noise map. The predicted noise map is the noise latent predicted by the target denoising model in the second initial noise map through the information coding characteristics of the object representation information.

[0220] Therefore, the generating device can perform a difference process on the second initial noise map and the predicted noise map to generate the object representation feature of the target object. For example, the second initial noise map can be used to subtract the predicted noise map (the elements at the same position in the second initial noise map and the predicted noise map can be subtracted) to obtain the object representation feature of the target object.

[0221] Optionally, the present application may also support setting up a specified LoRa module (a model parameter module, which can also be understood as a sub-model) in the target noise reduction model. The LoRa module can be used to generate customized personalized content (such as Disney effect content). Therefore, through the LoRa module, the target noise reduction model can generate object representation features containing the customized personalized content when performing object representation features, so that the target objects of different styles generated subsequently under the guidance of the object representation features can have the customized personalized content, such as the target object generated in the style image can be a Disney effect.

[0222] Among them, the lora module can be called the effect customization sub-model in the target noise reduction model. The specific customized personalized content of the lora module can be determined according to the actual training effect of the lora module, and this application does not limit this. In other words, the lora module can achieve corresponding control of the generation effect of the target object itself.

[0223] Since the object representation feature is obtained by the target denoising model after performing multiple rounds of denoising on the second initial noise map through the information encoding feature of the object representation information, the object representation feature can contain rich and complete content semantic-related features of the target object, and the target object can be fully characterized by the object representation feature. Therefore, by unifying the object representation feature as the input condition of each different style model to guide each style model to generate features, the style model can accurately generate high-quality style images with the same target object but different style types.

[0224] From the above, it can be seen that the method of the present application can include two processing stages (including the first stage and the second stage). The first stage is the stage of generating object representation features of the target object through the target denoising model, and the second stage is to guide each style model to generate style images of the same target object under multiple style types through the object representation features.

[0225] For example, the present application can be applied to but not limited to the following application scenarios:

[0226] Personalized AI (artificial intelligence) image generation scenario: This application can be applied to various personalized AI image generation scenarios. Users can input their requirements for image generation (which can be in text form, and the requirements can be used as object representation information of the target object) and select multiple style types (such as selecting multiple style prompt information). The method provided in this application can be used to generate AI images of multiple style types that are consistent with the user's requirements with one click.

[0227] Game promotion scenarios: This application can be applied to the promotion scenarios of game characters in various games. For example, the target object can be a game character in the game. The method provided in this application can be used to generate effect images or posters of various styles (i.e., style types) of the game character with one click. Then, according to the promotion needs of different promotion scenarios, the effect images or posters of different styles of the game character can be used to promote the game.

[0228] Promotion and marketing scenarios for film and television dramas: The present application can be applied to the promotion and marketing scenarios for various film and television dramas. For example, by using the method provided in the present application, it is possible to generate character images of a certain film and television character in a film and television drama in a variety of styles. The character images in different styles can be used to promote the film and television drama in different promotion and marketing scenarios, and the character images in different styles can also be used to make pillows, dolls, or building blocks and other related products for online sales.

[0229] See also Fig.10 , Fig.10 Schematic diagram of a scene for generating an object representation feature of a target object provided by an embodiment of the present application. Fig.10 As shown, the target denoising model may be a U-Net network, and the target denoising model may include a down sampling layer (ie, Down block), an intermediate sampling layer (ie, Mid block), and an up sampling layer (ie, Up block).

[0230] The object representation information of the target object may be the text representation information herein, and the generating device may perform feature encoding processing on the text representation information to generate a text embedding of the text representation information (i.e., an embedding feature of the text, which belongs to the above information encoding feature). The generating device may input the text embedding into the target denoising model to guide the target denoising model to generate the object representation feature of the target object. For example, the target denoising model may perform denoising processing on the second initial noise map under the cross attention of the text embedding to generate the object representation feature of the target object.

[0231] In another feasible implementation, the object representation information may also be information of an image type, such as an image of a target object, that is, the image may contain the target object. Similarly, the generating device may also use a trained image encoder (such as a VAE encoder) to perform feature encoding processing on the image type object representation information to generate image encoding features of the object representation information, and then the generating device may call the target denoising model to perform denoising processing on the second initial noise map under the constraints of the image encoding features to generate object representation features of the target object.

[0232] In this case, if the object representation information is image-type information, then correspondingly, the target denoising model can also be obtained by training with images related to the object, so that the target denoising model can learn and recognize features related to the image where the object is located. For example, the target denoising model can be obtained by training with a sample pair constructed by an object image and a standard image of the object in the object image. Optionally, the object image can be an image that a simulated user may input, which contains relevant training objects (can be any object that needs to be trained) but also has some other image elements (such as blue sky, trees, animations, icons, and other elements). The standard image can be a relatively clean and concise image where the target object is located, without a strong style type, and with no or few other image elements.

[0233] See also Fig.11 , Fig.11 is another schematic diagram of a scene for generating an object representation feature of a target object provided by an embodiment of the present application. Fig.11 As shown, the generating device can perform feature encoding processing on the object representation information of the target object, and can generate encoding features of the object representation information.

[0234] The generating device may call the target noise reduction model to perform noise reduction processing on the input second initial noise map under the constraint and guidance of the encoding feature of the object representation information to obtain a predicted noise map predicted by the second initial noise map. The generating device may perform a difference processing between the second initial noise map and the predicted noise map to obtain the object representation feature of the target object.

[0235] The specific form of the object representation information can be determined according to the actual application scenario, and this application does not impose any restrictions on this. Through the above process of this application, accurate and high-quality object representation features of the target object can be generated.

[0236] See also Fig.12 , Fig.12 Schematic diagram of the structure of an image generating device provided in an embodiment of the present application. Fig.12 As shown, the image generating device 120 may include: a first acquiring module 1201 , a second acquiring module 1202 , a feature generating module 1203 and an image generating module 1204 .

[0237] The first acquisition module 1201 is used to acquire an object representation feature of a target object, where the object representation feature is used to represent the content semantics of the target object;

[0238] The second acquisition module 1202 is used to acquire N style models and M style hint information associated with the N style models, where one style hint information is used to indicate the style semantics of a style type, and one style model is associated with one or more style hint information, and both N and M are positive integers, and N is less than or equal to M;

[0239] The feature generation module 1203 is used to call N style models to generate style image features corresponding to each style prompt information under the constraints of the object representation features and the associated style prompt information, wherein one style image feature includes the content features of the target object and the style features of the style type indicated by the corresponding style prompt information, wherein the content features are generated by performing feature learning on the content semantics represented by the object representation features, and the style features are generated by performing feature learning on the style semantics of the style type indicated by the corresponding style prompt information;

[0240] The image generation module 1204 is used to convert the style image features corresponding to the M style prompt information respectively to generate style images of the target object under M style types.

[0241] Optionally, any one of the N style models is a target style model, and any style hint information associated with the target style model is the target style hint information. The feature generation module 1203 calls the N style models to generate a style image feature corresponding to each style hint information under the constraints of the object representation feature and the associated style hint information, including:

[0242] The target style model is called to generate object constraint features of the target object under the constraints of the object representation features, and the object constraint features are used to reflect the content features of the target object;

[0243] Calling the target style model to generate style constraint features of the target object under the constraints of the target style prompt information, where the style constraint features are used to reflect the style features of the style type indicated by the target style prompt information;

[0244] The object constraint features and style constraint features are fused to generate style image features corresponding to the target style prompt information.

[0245] Optionally, the target style model includes K feature interaction sub-models connected in series for feature generation, and any feature interaction sub-model includes an object feature interaction branch and a style feature interaction branch connected in parallel, where K is a positive integer;

[0246] The target style model has a first initial noise map to be denoised, the K feature interaction sub-models include an i-th feature interaction sub-model, i is an integer, and i is less than or equal to K, and the i-th feature interaction sub-model has an input noise map;

[0247] If i is equal to 1, the input noise map is the first initial noise map. If i is greater than 1, the input noise map is the output feature of the i-1th feature interaction sub-model of the target style model. The output feature is obtained by fusing the feature generated by the object feature interaction branch in the i-1th feature interaction sub-model with the feature generated by the style feature interaction branch.

[0248] Optionally, the feature generation module 1203 calls the target style model to generate the object constraint feature of the target object under the constraint of the object representation feature, including:

[0249] Call the object feature interaction branch in the i-th feature interaction sub-model, perform denoising on the input noise map under the constraint of the object representation feature, and generate the object constraint sub-feature of the i-th feature interaction sub-model for the target object;

[0250] Determine the object constraint sub-feature of the target object for the Mth feature interaction sub-model of the target style model as the object constraint feature;

[0251] The process of performing denoising on the input noise map under the constraint of the object representation feature is a process of interactively processing the object representation feature and the input noise map.

[0252] Optionally, the feature generation module 1203 calls the target style model to generate the style constraint feature of the target object under the constraint of the target style prompt information, including:

[0253] Call the style feature interaction branch in the i-th feature interaction sub-model, perform denoising on the input noise map under the constraint of the target style prompt information, and generate the initial style constraint feature of the i-th feature interaction sub-model for the target object;

[0254] Based on the initial style constraint features, generate the style constraint sub-features of the i-th feature interaction sub-model for the target object;

[0255] Determine the style constraint sub-feature of the target object in the Mth feature interaction sub-model of the target style model as the style constraint feature;

[0256] Among them, the process of denoising the input noise map under the constraint of the target style hint information is the process of interactively processing the features of the target style hint information and the input noise map.

[0257] Optionally, the style feature interaction branch of the i-th feature interaction sub-model includes a feature controller, and the feature controller is used to control the fusion strength of the features generated by the style feature interaction branch of the i-th feature interaction sub-model when they are fused with the features generated by the object feature interaction branch of the i-th feature interaction sub-model;

[0258] The feature generation module 1203 generates the style constraint sub-feature of the i-th feature interaction sub-model for the target object based on the initial style constraint feature, including:

[0259] Based on the feature controller, the initial style constraint feature is controlled and processed to generate the style constraint sub-feature of the target object of the i-th feature interaction sub-model.

[0260] Optionally, the feature controller is configured as a switch control mode or a weight control mode, the switch control mode has an on state and an off state, and the weight control mode has an associated configured fusion weight;

[0261] The feature generation module 1203 controls and processes the initial style constraint feature based on the feature controller to generate the style constraint sub-feature of the i-th feature interaction sub-model for the target object, including:

[0262] If the feature controller is configured as a switch control mode, and the switch control mode is in an on state, the initial style constraint feature is determined as the style constraint sub-feature of the i-th feature interaction sub-model for the target object;

[0263] If the feature controller is configured as the switch control mode and the switch control model is in the off state, it is determined that the style constraint sub-feature of the target object of the i-th feature interaction sub-model is empty;

[0264] If the feature controller is configured in weight control mode, the initial style constraint feature is weighted based on the fusion weight to generate the style constraint sub-feature of the target object of the i-th feature interaction sub-model.

[0265] Optionally, the object constraint feature includes L object channel features on L feature channels, and the style constraint feature includes L style channel features on L feature channels, where L is a positive integer;

[0266] The feature generation module 1203 fuses the object constraint feature and the style constraint feature to generate a style image feature corresponding to the target style prompt information, including:

[0267] The object channel features and style channel features on the same feature channel are summed up respectively to generate the summed features of the target style model on each feature channel;

[0268] The object channel features and style channel features on the same feature channel are respectively concatenated in the first step to generate concatenated features of the target style model on each feature channel;

[0269] Combine the summed features and concatenated features of the target style model on L feature channels to generate style image features corresponding to the target style prompt information;

[0270] Among them, the fusion processing includes addition processing, the first step of splicing processing and combination processing.

[0271] Optionally, the feature generation module 1203 combines the summed features and the concatenated features of the target style model on L feature channels to generate a style image feature corresponding to the target style prompt information, including:

[0272] The summed features and concatenated features of the target style model on the same feature channel are concatenated in the second step to generate transition fusion features of the target style model on each feature channel;

[0273] Transform the transition fusion features of the target style model on L feature channels to generate style image features corresponding to the target style prompt information;

[0274] The combined processing includes the second step of splicing processing and transformation processing.

[0275] Optionally, the feature generation module 1203 transforms the transition fusion features of the target style model on L feature channels to generate style image features corresponding to the target style prompt information, including:

[0276] Get the convolution model; the convolution model contains L convolution kernels;

[0277] L convolution kernels are used to perform convolution processing on the transition fusion features of the target style model on L feature channels respectively, to generate convolution features of the target style model on L feature channels, and one convolution kernel is used to perform convolution processing on the transition fusion features of the target style model on one feature channel;

[0278] Call the convolution model to merge the convolution features of the target style model on L feature channels to generate style image features corresponding to the target style prompt information;

[0279] Among them, the transformation processing includes convolution processing and merging processing.

[0280] Optionally, the image generation module 1204 converts the M style image features corresponding to the M style prompt information respectively to generate style images of the target object under M style types, including:

[0281] Call the image decoder to decode each style image feature separately to generate the style image of the target object under M style types;

[0282] The style image feature corresponding to a style hint information is used to decode and generate a style image of a target object under a style type, and the conversion process includes a decoding process.

[0283] Optionally, the first acquisition module 1201 acquires the object representation feature of the target object in a manner including:

[0284] Obtaining object representation information of the target object and a second initial noise map to be denoised;

[0285] The target denoising model is called to perform denoising on the second initial noise map under the constraint of the object representation information to generate object representation features.

[0286] Optionally, the first acquisition module 1201 calls the target denoising model to perform denoising on the second initial noise image under the constraint of the object representation information to generate the object representation feature, including:

[0287] Calling the target denoising model to perform denoising on the second initial noise map under the constraint of the object representation information, and generating a predicted noise map of the target denoising model for the second initial noise map;

[0288] The second initial noise map and the predicted noise map are subjected to difference processing to generate object representation features.

[0289] Optionally, the object representation information is obtained by parsing an image generation request sent by an image client, the image generation request further carries M style prompt information, the image client includes H style prompt information that can be selected, the H style prompt information includes a full amount of style prompt information associated with a full amount of style models, the M style prompt information is selected from the H style prompt information, H is a positive integer, and H is greater than or equal to M;

[0290] The image generation module 1204 is also used for:

[0291] Return the style images of the target object under M style types to the image client;

[0292] The image client is used to associate and display the style images of the target object under M style types with the corresponding style prompt information respectively.

[0293] According to one embodiment of the present application, Figure 3 The steps involved in the image generation method shown can be represented by Fig.12 The various modules in the image generating device 120 shown in the figure are executed. For example, Figure 3 The step S101 shown in FIG. 1 can be performed by Fig.12 The first acquisition module 1201 in is executed, Figure 3 The step S102 shown in FIG. 1 can be performed by Fig.12The second acquisition module 1202 is executed; Figure 3 The step S103 shown in FIG. 1 can be performed by Fig.12 The feature generation module 1203 is used to execute, Figure 3 The step S104 shown in FIG. 1 can be performed by Fig.12 The image generation module 1204 in is executed.

[0294] The present application can obtain the object representation feature of the target object, the object representation feature is used to characterize the content semantics of the target object; and can obtain N style models and M style hint information associated with the N style models, one style hint information is used to indicate the style semantics of a style type, one style model is associated with one or more style hint information, N and M are both positive integers, and N is less than or equal to M; thus, the N style models can be called to generate style image features corresponding to each style hint information under the constraints of the object representation feature and the associated style hint information, one style image feature includes the content feature of the target object and the style feature of the style type indicated by the corresponding style hint information, the content feature is generated by feature learning of the content semantics represented by the object representation feature, and the style feature is generated by feature learning of the style semantics of the style type indicated by the corresponding style hint information; and the M style image features corresponding to the M style hint information can be converted respectively to generate a style image of the target object under M style types. It can be seen that the method proposed in the present application guides each style model to generate style image features associated with each style hint information through the same object representation features of the target object and the style hint information associated with each style model. Each style image feature can include not only content features learned from the content semantics of the target object, but also style features learned from the style semantics of the style type indicated by the corresponding style hint information. Therefore, high-quality style images of the target object in various style types can be quickly generated through each style image feature. Each style image can contain the content of the same target object, but the content of the target object can have different style types in each style image. Therefore, the generation efficiency of the style image of the target object is improved, and the high quality of the style image of the target object is guaranteed.

[0295] According to one embodiment of the present application, Fig.12The various modules in the image generating device 120 shown can be separately or completely combined into one or several units to constitute, or one (some) of the units can be further divided into multiple functionally smaller sub-units, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above modules are divided based on logical functions. In practical applications, the functions of one module can also be implemented by multiple units, or the functions of multiple modules can be implemented by one unit. In other embodiments of the present application, the image generating device 120 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0296] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0297] According to one embodiment of the present application, a computer program capable of executing the steps involved in the corresponding methods shown in the embodiments of the present application can be run on a general-purpose computer device (the computer device may include processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM)) to construct the following. Fig.12 The image generating device 120 shown in FIG. The computer program may be recorded on a computer-readable recording medium, and may be loaded into the computer device through the computer-readable recording medium and run therein.

[0298] See also Fig.13 , Fig.13 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.13As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, in some embodiments, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. As Fig.13 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0299] exist Fig.13 In the computer device 1000 shown in the figure, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the computer program stored in the memory 1005, so that the computer device 1000 described in the embodiments of the present application can execute the description of the above-mentioned image generation method in each embodiment of the present application, and can also execute the above-mentioned Fig.12 The description of the image generating device 120 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0300] In addition, it should be pointed out here that: the present application also provides a computer-readable storage medium, and a computer program is stored in the computer-readable storage medium. When the processor executes the computer program, the description of the image generation method in each embodiment of the present application can be executed, so it will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer storage medium embodiment involved in the present application, please refer to the description of the method embodiment of the present application.

[0301] As an example, the above-mentioned computer program may be deployed on one computer device for execution, or deployed on multiple computer devices located at one location for execution, or executed on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain network.

[0302] The computer-readable storage medium may be an internal storage unit of the computer device, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Further, the computer-readable storage medium may include both an internal storage unit and an external storage device of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0303] The present application provides a computer program product, which includes a computer program, which is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the description of the above-mentioned image generation method in each embodiment of the present application, and therefore, it will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in the present application, please refer to the description of the method embodiment of the present application.

[0304] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0305] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0306] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire an object representation feature of a target object, wherein the object representation feature is used to characterize the content semantics of the target object, and the object representation feature is generated by calling a target denoising model and using the object representation information of the target object as a constraint condition for denoising to perform denoising on the second initial noise image, wherein the target denoising model belongs to a diffusion model; Obtaining N style models and M style hint information associated with the N style models, wherein one piece of the style hint information is used to indicate the style semantics of a style type, and one style model is associated with one or more of the style hint information, N and M are both positive integers, and N is less than or equal to M, and the N style models all belong to diffusion models; Calling the N style models to generate a style image feature corresponding to each style hint information under the constraints of the object representation feature and the associated style hint information, wherein one style image feature includes a content feature of the target object and a style feature of a style type indicated by the corresponding style hint information, wherein the content feature is generated by performing feature learning on the content semantics represented by the object representation feature, and the style feature is generated by performing feature learning on the style semantics of the style type indicated by the corresponding style hint information, and the object representation feature and the M style hint information are constraints for the N style models to perform denoising processing to generate style image features; The style image features corresponding to the M style prompt information are converted respectively to generate style images of the target object under M style types.

2. The method according to claim 1, characterized in that Any one of the N style models is a target style model, any style hint information associated with the target style model is the target style hint information, and calling the N style models to generate a style image feature corresponding to each style hint information under the constraints of the object representation feature and the associated style hint information includes: Calling the target style model to generate an object constraint feature of the target object under the constraint of the object representation feature, wherein the object constraint feature is used to reflect the content feature of the target object; Calling the target style model to generate a style constraint feature of the target object under the constraint of the target style prompt information, wherein the style constraint feature is used to reflect the style feature of the style type indicated by the target style prompt information; The object constraint feature and the style constraint feature are fused to generate a style image feature corresponding to the target style prompt information.

3. The method according to claim 2, characterized in that The target style model includes K feature interaction sub-models connected in series for feature generation, and any feature interaction sub-model includes an object feature interaction branch and a style feature interaction branch connected in parallel, where K is a positive integer; The target style model has a first initial noise map to be denoised, the K feature interaction sub-models include an i-th feature interaction sub-model, i is an integer, and i is less than or equal to K, and the i-th feature interaction sub-model has an input noise map; If i is equal to 1, then the input noise map is the first initial noise map. If i is greater than 1, then the input noise map is the output feature of the i-1th feature interaction sub-model of the target style model, and the output feature is obtained by fusing the feature generated by the object feature interaction branch in the i-1th feature interaction sub-model with the feature generated by the style feature interaction branch.

4. The method according to claim 3, characterized in that The calling of the target style model to generate the object constraint feature of the target object under the constraint of the object representation feature includes: Calling the object feature interaction branch in the i-th feature interaction sub-model, performing denoising on the input noise map under the constraint of the object representation feature, and generating an object constraint sub-feature of the i-th feature interaction sub-model for the target object; Determine the object constraint sub-feature of the target object for the Mth feature interaction sub-model of the target style model as the object constraint feature; The process of performing denoising on the input noise map under the constraint of the object representation feature is a process of performing feature interaction processing on the object representation feature and the input noise map.

5. The method according to claim 3, characterized in that The calling of the target style model to generate the style constraint feature of the target object under the constraint of the target style prompt information includes: Calling the style feature interaction branch in the i-th feature interaction sub-model, performing denoising on the input noise map under the constraint of the target style prompt information, and generating an initial style constraint feature of the i-th feature interaction sub-model for the target object; Based on the initial style constraint feature, generating a style constraint sub-feature of the i-th feature interaction sub-model for the target object; Determining the style constraint sub-feature of the target object in the Mth feature interaction sub-model of the target style model as the style constraint feature; The process of performing denoising on the input noise map under the constraint of the target style prompt information is a process of performing feature interaction processing on the target style prompt information and the input noise map.

6. The method according to claim 5, characterized in that The style feature interaction branch of the i-th feature interaction sub-model includes a feature controller, and the feature controller is used to control the fusion strength of the features generated by the style feature interaction branch of the i-th feature interaction sub-model when the features are fused with the features generated by the object feature interaction branch of the i-th feature interaction sub-model; The step of generating the style constraint sub-feature of the i-th feature interaction sub-model for the target object based on the initial style constraint feature includes: Based on the feature controller, the initial style constraint feature is controlled and processed to generate a style constraint sub-feature of the i-th feature interaction sub-model for the target object.

7. The method according to claim 6, characterized in that The feature controller is configured as a switch control mode or a weight control mode, wherein the switch control mode has an on state and an off state, and the weight control mode has an associated configured fusion weight; The controlling and processing the initial style constraint feature based on the feature controller to generate the style constraint sub-feature of the i-th feature interaction sub-model for the target object includes: If the feature controller is configured as the switch control mode, and the switch control mode is in the on state, the initial style constraint feature is determined as the style constraint sub-feature of the i-th feature interaction sub-model for the target object; If the feature controller is configured as the switch control mode, and the switch control mode is in an off state, determining that the style constraint sub-feature of the i-th feature interaction sub-model for the target object is empty; If the feature controller is configured in the weight control mode, the initial style constraint feature is weighted based on the fusion weight to generate the style constraint sub-feature of the i-th feature interaction sub-model for the target object.

8. The method according to claim 2, characterized in that The object constraint feature includes L object channel features on L feature channels, and the style constraint feature includes L style channel features on the L feature channels, where L is a positive integer; The fusing process of the object constraint feature and the style constraint feature to generate a style image feature corresponding to the target style prompt information includes: The object channel features and the style channel features on the same feature channel are respectively summed to generate the summed features of the target style model on each feature channel; Performing the first step of splicing processing on the object channel features and the style channel features on the same feature channel respectively, to generate splicing features of the target style model on each feature channel; Combining the summed features and concatenated features of the target style model on the L feature channels to generate a style image feature corresponding to the target style prompt information; The fusion process includes the addition process, the first step of splicing process and the combination process.

9. The method according to claim 8, characterized in that The combining and processing the summed features and the concatenated features of the target style model on the L feature channels to generate a style image feature corresponding to the target style prompt information includes: Performing a second step of splicing processing on the summed features and the spliced ​​features of the target style model on the same feature channel, respectively, to generate a transition fusion feature of the target style model on each feature channel; Transforming the transition fusion features of the target style model on the L feature channels to generate style image features corresponding to the target style prompt information; The combination process includes the second-step splicing process and the transformation process.

10. The method according to claim 9, characterized in that The transforming process is performed on the transition fusion features of the target style model on the L feature channels to generate style image features corresponding to the target style prompt information, including: Obtain a convolution model; the convolution model includes L convolution kernels; Using the L convolution kernels to perform convolution processing on the transition fusion features of the target style model on the L feature channels respectively, to generate convolution features of the target style model on the L feature channels, one convolution kernel is used to perform convolution processing on the transition fusion features of the target style model on one feature channel; Calling the convolution model to merge the convolution features of the target style model on the L feature channels to generate a style image feature corresponding to the target style prompt information; The transformation process includes the convolution process and the merging process.

11. The method according to claim 1, characterized in that The converting the style image features corresponding to the M style prompt information to generate the style image of the target object under the M style types includes: Calling an image decoder to decode each style image feature respectively, to generate a style image of the target object under the M style types; Wherein, a style image feature corresponding to the style hint information is used to decode and generate a style image of the target object under a style type, and the conversion process includes the decoding process.

12. The method according to claim 1, characterized in that The step of obtaining the object representation feature of the target object includes: Acquire the object representation information of the target object and the second initial noise map to be denoised; The target denoising model is called to perform denoising processing on the second initial noise map under the constraint of the object representation information to generate the object representation feature.

13. The method according to claim 12, characterized in that The calling target denoising model performs denoising processing on the second initial noise map under the constraint of the object representation information to generate the object representation feature, including: Calling the target denoising model to perform denoising on the second initial noise map under the constraint of the object representation information, and generating a predicted noise map of the target denoising model for the second initial noise map; The second initial noise map and the predicted noise map are subjected to difference processing to generate the object representation feature.

14. The method according to claim 11, characterized in that The object representation information is obtained by parsing an image generation request sent by an image client, the image generation request further carries the M style prompt information, the image client includes H style prompt information that can be selected, the H style prompt information includes a full amount of style prompt information associated with a full amount of style models, the M style prompt information is selected from the H style prompt information, H is a positive integer, and H is greater than or equal to M; The method further comprises: Returning the style images of the target object under the M style types to the image client; The image client is used to associate and display the style images of the target object under the M style types with the corresponding style prompt information respectively.

15. An image generating device, characterized in that: The device comprises: A first acquisition module is used to acquire an object representation feature of a target object, wherein the object representation feature is used to characterize the content semantics of the target object, and the object representation feature is generated by calling a target denoising model and using the object representation information of the target object as a constraint condition for denoising to perform denoising on the second initial noise image, wherein the target denoising model belongs to a diffusion model; A second acquisition module is used to acquire N style models and M style hint information associated with the N style models, wherein one piece of the style hint information is used to indicate the style semantics of a style type, and one style model is associated with one or more of the style hint information, N and M are both positive integers, and N is less than or equal to M, and the N style models are all diffusion models; A feature generation module, used for calling the N style models to generate a style image feature corresponding to each style hint information under the constraints of the object representation feature and the associated style hint information, wherein one style image feature includes a content feature of the target object and a style feature of a style type indicated by the corresponding style hint information, wherein the content feature is generated by performing feature learning on the content semantics represented by the object representation feature, and the style feature is generated by performing feature learning on the style semantics of the style type indicated by the corresponding style hint information, and the object representation feature and the M style hint information are constraints for the N style models to perform denoising processing to generate style image features; The image generation module is used to convert the style image features corresponding to the M style prompt information respectively to generate style images of the target object under M style types.

16. A computer program product, characterized in that The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 14 when being executed by a processor.

17. A computer device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 14.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps of the method according to any one of claims 1 to 14.

Images