Image generation method

Abstract

The embodiment of the invention provides an image generation method, and the method comprises the steps: obtaining a user portrait of a target user based on an image generation service request initiated by the target user; performing matching processing on the user portrait and a preset image style template library through a user style portrait prediction model to obtain a user style portrait prediction result of the target user; generating an image generation template drawing of the target user according to the user style portrait prediction result and the preset image style template library through an image generation model; and through the image generation model, generating a target image of the target user according to the image generation template image and the target image reference image. Through the embodiment of the invention, the problem that a personalized image conforming to the preferences and features of the user cannot be automatically generated according to the preferences and features of the user, the face features and the like in the related technology can be solved, so that the generated target image is more accurate and better fits the preferences and features of the target user.

Description

Technical Field

[0001] The present application relates to the technical field of image generation, and in particular to an image generation method. Background Art

[0002] With the development of computer technology, image generation technology has been widely used in various fields. Through image generation technology, user groups can quickly and cheaply obtain ideal images. In the scenario where users use applications, users can usually set a personalized image that identifies the user's identity in the user avatar frame of the application when using an application with a personalized setting function. However, the image setting function of existing applications only supports user photo upload, album photo selection, and preset cartoon, illustration and other selection settings. It is impossible to automatically generate personalized images that meet the user's preferences and characteristics based on the user's preferences, characteristics, and facial features, resulting in the user being unable to set a personalized avatar that meets their own preferences and characteristics. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an image generation method to solve the problem in the related art that personalized images that meet user preferences and characteristics cannot be automatically generated.

[0004] In order to solve the above technical problems, the embodiments of the present application are implemented as follows: In a first aspect, an embodiment of the present application provides an image generation method, comprising: Based on the image generation service request initiated by the target user, a user portrait of the target user is obtained; The user style portrait prediction model is used to match the user portrait with a preset image style template library to obtain a user style portrait prediction result of the target user; Generate an image generation template graph of the target user according to the user style portrait prediction result and the preset image style template library through the image generation model; The target image of the target user is generated by the image generation model according to the image generation template image and the target image reference image.

[0005] In a second aspect, an embodiment of the present application provides an image generating device, including: An acquisition module, configured to generate a service request based on an image initiated by a target user and acquire a user portrait of the target user; A prediction module is used to match the user portrait with a preset image style template library through a user style portrait prediction model to obtain a user style portrait prediction result of the target user; A first generation module is used to generate an image generation template graph of the target user according to the user style portrait prediction result and the preset image style template library through an image generation model; The second generation module is used to generate a target image of the target user through the image generation model according to the image generation template image and the target image reference image.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect above are implemented.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the steps of the method described in the first aspect above.

[0009] It can be seen from the technical solution provided by the above embodiment of the present application that, in the embodiment of the present application, first, based on the image generation service request initiated by the target user, the user portrait of the target user is obtained; through the user style portrait prediction model, the user portrait and the preset image style template library are matched and processed to obtain the user style portrait prediction result of the target user; through the image generation model, the image generation template graph of the target user is generated according to the user style portrait prediction result and the preset image style template library; through the image generation model, the target image of the target user is generated according to the image generation template graph and the target image reference graph. It can be seen that through the embodiment of the present application, the user style portrait of the target user can be determined based on the user portrait of the target user and the preset image style template library through the user style portrait prediction model, and then the image generation template graph of the target user is generated through the image generation model, so that the target image of the target user can be generated according to the image generation template graph and the target image reference graph, which solves the problem that the related technology cannot automatically generate personalized images that meet the user's preferences and characteristics according to the user's preferences, characteristics and facial features, etc., so that the generated target image is more accurate and more in line with the preferences and characteristics of the target user, and the user experience is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0011] Figure 1 A schematic diagram of a flow chart of an image generation method provided in an embodiment of the present application; Figure 2 A schematic diagram of the working process of the user style portrait prediction model provided in the embodiment of the present application; Figure 3 A schematic diagram of an example prompt for uploading an image provided in an embodiment of the present application; Figure 4 A schematic diagram of the architecture of the image generation model provided in the embodiment of the present application; Figure 5 A schematic diagram of the architecture of the Adapter module and the Adapter fusion module provided in the embodiment of the present application; Figure 6 A schematic diagram of a flow chart of another image generation method provided in an embodiment of the present application; Figure 7 A schematic diagram of the module composition of an image generating device provided in an embodiment of the present application; Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0012] An embodiment of the present application provides an image generating method.

[0013] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0014] Before introducing the image generation method provided in the embodiment of the present application, the application scenario provided in the embodiment of the present application is described first. The image generation method provided in the embodiment of the present application can be applied to the scenario where user A sets an avatar in application software A. In this scenario, the application can be a tool application such as a cloud disk, a mailbox, or a social media software such as WeChat and Weibo, and this application does not limit this. The application is installed in the terminal device used by user A, and the terminal device can be an electronic device such as a mobile phone, a tablet computer, a laptop computer, a PDA, a car-mounted electronic device, etc.

[0015] The image generation method provided in the embodiment of the present application is clearly and completely described below in combination with the above application scenarios and the drawings in the embodiment of the present application.

[0016] Figure 1 The present invention provides a flowchart of an image generation method provided in an embodiment of the present invention. The method can be executed by a server, which can be an independent server, a server cluster composed of multiple servers, or a cloud server for cloud computing processing. Figure 1 As shown, the method may specifically include the following steps: Step S102: Generate a service request based on the image initiated by the target user to obtain a user portrait of the target user.

[0017] In the embodiment of the present application, the image generation service request initiated by the target user may be initiated when the target user sets an avatar in the application. User portrait, also known as user information labeling or customer labeling, is a highly refined user model constructed by collecting and analyzing user social attributes, living habits, consumption behavior and other information. User portrait includes user basic information, user behavior data, user preference information and user social information.

[0018] Among them, basic user information includes user age, gender, occupation and other information; user behavior data includes user browsing, clicking, purchasing, sharing, commenting, publishing and other behaviors, reflecting user interests, needs, habits and behavior patterns; user preference information includes user preferences, attitudes, values, emotions and other information, reflecting user personality and style; user social information includes user friends, attention, fans, communities, groups and other information, reflecting user social relationships and influence. Among them, basic user information belongs to static data, which can be obtained through user account registration information extraction, or obtained from other third-party platforms based on user account mobile phone number; user behavior, preferences and social information are dynamic data, which can be extracted from dynamic data generated by users using social and shopping platforms in a certain period of time, including cloud disk circles, WeChat, Douyin, Mafengwo, Weibo, NetEase Cloud Music, Taobao, JD.com, Dianping, etc., and can also be extracted from questionnaires, evaluations and feedback information of various platforms in which users participate. This application does not impose specific restrictions on this.

[0019] Step S104: using the user style portrait prediction model, matching the user portrait with the preset image style template library to obtain a user style portrait prediction result of the target user.

[0020] In the embodiment of the present application, the preset style template library may be a user avatar style type template library preset in the application, and the preset style template library may include multiple style type templates, for example, the style type may include "career elite, ancient style fairy, two-dimensional, happy new year, hairpin flower", etc. Among them, the images in the preset style template library may have the following sources: one is the artistic shooting of real models; the other is the design and drawing of designers.

[0021] In an embodiment of the present application, the acquired user portrait and the preset image style template library are input into a user style portrait prediction model, and a user style portrait prediction result of the target user is obtained through the user style portrait prediction model.

[0022] In one embodiment, the user style portrait prediction model is used to match the user portrait with a preset image style template library to obtain a user style portrait prediction result of the target user, including: Determine the matching degree between the user portrait and each template image in the preset image style template library through the user style portrait prediction model; Through the user style portrait prediction model, the user style portrait prediction result of the target user is determined according to the matching degree between the user portrait and each template image in the preset image style template library.

[0023] In the embodiment of the present application, the acquired user portrait and the preset image style template library are input into the user style portrait prediction model, and the user style portrait prediction model is used to calculate the matching degree between the user portrait and each template image in the preset image template library. Then, according to the matching degree between the user portrait and each template image in the preset image style template library, the user style portrait prediction result of the target user is determined.

[0024] In the embodiment of the present application, the matching degree between the user portrait calculated by the user style portrait prediction model and each template image in the preset image template library can be represented by a matching degree vector.

[0025] In one embodiment, determining the degree of match between the user portrait and each template image in the preset image style template library through the user style portrait prediction model includes: Through the user style portrait prediction model, the user portrait of the target user is processed with characteristic keywords to obtain the characteristic keywords of the target user; Through the user style portrait prediction model, the feature keywords and prompt information are encoded to obtain the user portrait text features; the prompt information includes the style category of the template image and the context information representing the style of the template image; Through the user style portrait prediction model, feature extraction is performed on each template image in the preset image style template library to obtain the representation features of the template image; Through the user style portrait prediction model, the matching degree between the user portrait and each template image is determined according to the text features of the user portrait and the representation features of each template image.

[0026] In one implementation, four types of user information in the user portrait of the target user are input into a user style portrait prediction model, and the user basic information, user behavior data, user preference information, and user social information in the user portrait are processed through the user style portrait prediction model to perform feature keyword extraction processing to obtain the corresponding feature keywords of the target user. The feature keywords of the four types of target users can be represented by {K1, K2, K3, K4}, respectively.

[0027] In another embodiment, user characteristic keywords can also be extracted through a text generation model. The four types of user information in the user portrait are input into the text generation model, and the text generation model is used to generate user characteristic keywords for each type of user information to obtain characteristic keywords {K1, K2, K3, K4} of the target user.

[0028] After obtaining the characteristic keywords of the target user, the characteristic keywords and prompt information of the target user are input into the user style portrait prediction model, and the characteristic keywords and prompt information are encoded by the style portrait prediction model to obtain the user portrait text features; wherein the prompt information includes the style category CLASS of the template image and the context information characterizing the style of the template image.

[0029] Next, the preset style template library is input into the user style portrait prediction model. Through the user style portrait prediction model, feature extraction processing is performed on each template image in the preset style template library to obtain the representation features of each template image.

[0030] Then, the user portrait text features and the representation features of each template image are input into the user style portrait prediction model, and the matching degree between the user portrait text features and each template image is calculated through the user style portrait prediction model.

[0031] In an exemplary implementation, a user feature keyword condition embedding attention mechanism is used in a user style portrait prediction model to obtain a more accurate user style portrait prediction result. Figure 2 A schematic diagram of the working process of the user style portrait prediction model provided in the embodiment of the present application is shown in FIG. Figure 2 As shown in the figure, the user style portrait prediction model includes a set of dynamically learnable context vectors {p1, p2, ... pM}, a text encoder that receives the target user's characteristic keywords as controllable condition input, and a pre-trained image encoder with fixed parameters. CLASS corresponds to the style category token of the template image in the preset style template library. The style category of the template image is, for example, "career elite", "ancient fairy", "secondary element", "Happy New Year", "hairpin", etc. The role of the learnable context vector {p1, p2, ... pM} is to provide a set of dynamically learnable prompt information for the text encoder, prompting the text encoder to learn more features related to the template image. The learnable context vector and the preset style template library category token are spliced ​​and input into the text encoder as K and V components after splicing. The characteristic keyword is input into the text encoder as the Q component, and the user characteristic keyword embedding is completed through the cross attention layer calculation, thereby obtaining the user style portrait text feature St based on the characteristic keyword.

[0032] Specifically, assume that the learnable context vector and the template image style category token are concatenated to obtain the vector Ct=[p]1[p]2...[p]m[CLASS], the feature keyword encoding vector is Ut=Embed([K1,K2,K3,k4]), the feature keyword conditional embedding attention submodule calculates the output result as t, the first self-attention layer calculates the output result as S1, and the second cross-attention layer calculates the output result as c2. FEDW(·) and LN(·) are used to represent the Feedforward and LayerNorm layer functions respectively, then: ; (1) ; (2) ; (3) Finally, after N feature keyword conditions are embedded into the attention submodule, the user style portrait text feature St is calculated. Figure 2 As shown in FIG. 1 , the feature keyword encoding vector Ut is input into the cross attention layer of the N user feature keyword condition embedding attention submodule, and the calculation process of formula (1) to formula (3) is completed to calculate the user style portrait text feature St.

[0033] The user style portrait prediction learning is completed by matching the user style portrait text feature St with each template image in the preset style template library. Specifically, each template image in the preset style template library is passed through a pre-trained image encoder to extract the representation feature It of each template image. The pre-trained image encoder can be, for example, CLIPImageEncoder (image encoder).

[0034] In one embodiment, the user style portrait prediction model is used to determine the user style portrait prediction result of the target user according to the matching degree between the user portrait and each template image in the preset image style template library, including: Through the user style portrait prediction model, each matching degree between the user portrait and each template image in the preset image style template library is determined as the user style portrait prediction result of the target user.

[0035] In an embodiment of the present application, after determining each matching degree between the user portrait and each template image in the preset image style template library, each matching degree between the user portrait and each template image in the preset image style template library is determined as a user style portrait prediction result of the target user.

[0036] Continuing with the above example, the user portrait style prediction model calculates the matching degree vector between the user style portrait text feature St and the representation feature It of each template image, and outputs the matching degree vector W1r between each template image and the user style portrait as the user style portrait prediction result. Where W1r{r=1,2,..R}, r is the rth template image in the preset style template library, and R is the total number of template images in the preset style template library.

[0037] In addition, the user portrait style prediction model will also calculate the maximum match between the user style portrait text feature St and each template image representation feature It in the preset style template library. The maximum match is the best user style portrait. In addition, the user portrait style prediction model continuously adjusts parameters during the training phase to make the output user style portrait more accurate.

[0038] Step S106, using the image generation model, based on the user style portrait prediction result and the preset image style template library, an image generation template graph of the target user is generated.

[0039] In an embodiment of the present application, after obtaining the user style portrait prediction result, the user style portrait prediction result and the preset image style template library are input into the image generation model, and the image generation template graph of the target user is generated through the image generation model.

[0040] In one embodiment, an image generation template graph of a target user is generated by an image generation model according to the user style portrait prediction result and a preset image style template library, including: Get the weight of each template image in the preset image style template library; Through the image generation model, the user style portrait prediction results and each template image in the preset image style template library are weighted to obtain the image generation template map of the target user.

[0041] In an embodiment of the present application, first, the weight of each template image in the preset image style template library is obtained, and then the user style portrait prediction result and the preset image style template library are input into the image generation model. Through the image generation model, the user style portrait prediction result and each template image in the preset image style template library are weighted and summed to obtain the image generation template map of the target user.

[0042] In an exemplary implementation, assuming that the application is a cloud disk, when obtaining the weight of each template image in the preset image style template library, first, the cloud disk avatar marketing library table corresponding to the time node when user A initiates the image generation request can be queried. The avatar marketing library table is a list of style IDs that are mainly promoted by the operation activities during the time node when user A initiates the image generation request. Assuming that the time node when user A initiates the image generation request falls during the Spring Festival, the marketing library table style ID may be Happy New Year, Fireworks in Ten Thousand Families, and Ruyi Shi, etc.

[0043] Generate a template image weight vector based on marketing activity information by querying the avatar marketing library table , as shown in formula (4): (4) In formula (4), r is the rth template image in the preset style template library, r=1,2,..R; R is the total number of template images in the preset style template library, It is the weight vector corresponding to each template image in the preset style template library.

[0044] The weight vector corresponding to each template image The calculation method of is shown in formula (5) and formula (6): (5) (6) In formula (5), T is the total duration of the marketing promotion of template image r (in days or hours), t is the time (in days or hours) from the start of the promotion of template image r to the present, k is the number of times the template image r has been selected since the promotion, and z is the total number of times all template images have been selected since the promotion. For template images that are not in the marketing library table, their weight components are assigned a value of 0.

[0045] In formula (6), α and β are coefficients related to the image generation model and are used to adjust the output of the image generation model.

[0046] Next, the matching degree vector of each template image obtained above and the user style portrait is and the weight vector of the template image A weighted summation process is performed, and according to the weighted summation result, an image generation template graph of the target user is determined.

[0047] In one embodiment, the image generation model is used to perform weighted processing on the user style portrait prediction result and each template image in the preset image style template library to obtain an image generation template graph of the target user, including: Through the image generation model, the user style portrait prediction results and the weight of each template image are weighted to obtain the recommendation weight of each template image; Sorting the template images according to the recommendation weight of each template image to generate a template image list; In response to the target user triggering an operation of selecting a template image in the template image list, the template image selected by the target user is acquired, and the template image is determined as an image generation template image of the target user.

[0048] In the embodiment of the present application, first, the user style portrait prediction result and the weight of each template image are input into the image generation model, and the user style portrait prediction result and the weight of each template image are weighted and summed through the image generation model to obtain the recommended weight of each template image. Then, according to the recommended weight of each template image, the template images are sorted to generate a list of template images arranged in the order of recommended weights. Next, in response to the target user triggering the operation of selecting a template image in the template image list, the template image selected by the target user is obtained, and the template image is determined as the image generation template image of the target user.

[0049] In an exemplary implementation, the matching degree vector between each template image obtained in the above step and the user style portrait is And the weight vector of each template image The weighted sum is performed to obtain the recommended weight Score of each template image. Each template image is sorted according to the recommended weight Score to obtain a sorted template image list. When user A selects a template image, the list pops up for user A to select a template image. The template image selected by user A is the template image generated for user A's image.

[0050] The recommended weight Score of the template image can be obtained in the following ways: ; (7) In the above formula (7), Represents weight.

[0051] In another exemplary implementation, after sorting each template image according to the size of the recommendation weight Score to obtain a sorted template image list, if user A does not select a template image, then the top X template images are used as the image generation template image of user A by default, where the value of X can be 1, 2, for example.

[0052] Step S108: Generate a target image of a target user by using an image generation model to generate a template image and a target image reference image according to the image.

[0053] In the embodiment of the present application, the above-generated image generation template image and the target image reference image are input into the image generation model, and the target image of the target user is generated through the image generation model. The target image reference image can be an image with the same or similar features as the target image finally generated.

[0054] In one embodiment, the target image reference graph includes a face image of a target user; and generating a target image of the target user by using an image generation model and generating a template graph and a target image reference graph according to the image, including: Through the image generation model, feature extraction is performed on the target user's face image to obtain the target user's face features; Through the image generation model, the face mask features of the image generation template image and the face features of the target user are fused to obtain the target image of the target user.

[0055] In the embodiment of the present application, the target image reference image may be a face image of a target user. When the target image of the target user is generated by the image generation model according to the image generation template image and the target image reference image, first, the face image of the target user may be input into the image generation model, and feature extraction processing may be performed to obtain the face features of the target user. Then, the face mask features of the image generation template image and the face features of the target user are input into the image generation model, and feature fusion processing may be performed to finally generate the target image of the target user.

[0056] In one implementation, the face image of the target user may be a face image that the target user has taken in advance and uploaded to the application. Before the face image of the target user is subjected to feature extraction processing through the image generation model, a face image pre-uploaded by the user is received, and then the face image is subjected to image quality detection. The quality detection of the face image uploaded by the user includes face detection, clarity, occlusion, etc. In addition to detection through the image generation model, it can also be implemented through a preset face detection and face quality detection model.

[0057] If it is detected that the face image meets the quality requirements, the face image uploaded by the user is used as the target image reference image. If it is detected that the face image does not meet the quality requirements, other identical or similar face images uploaded by the user to the application are searched, and the face image that meets the requirements is selected as the target image reference image.

[0058] Specifically, when the face image uploaded by the user fails the quality inspection, first, a matching search is performed among the face images uploaded by the user to the application, and a set of identical or similar face images is obtained by matching, and a face image that meets the requirements is screened using a preset face quality inspection model as a target image reference. If the image uploaded by the user is a faceless or multi-face image, the matching search process in the application is skipped, and the user is directly prompted to re-upload the face image until a reference image that meets the requirements is obtained. When prompting the user to re-upload the face image, an example of the required image can be provided to the user.

[0059] Figure 3 A schematic diagram of an example prompt for uploading an image provided in an embodiment of the present application, such as Figure 3 As shown in the figure, the prompt image shows correct examples and incorrect examples of image upload. The correct example is the image corresponding to the "frontal photo" in the figure, and the incorrect examples are the images corresponding to the "face is blocked", "head is too small" and "group photo" in the figure. By showing the target user the image upload example prompt, the user is guided to upload a face image that meets the requirements.

[0060] In one implementation, after acquiring the face image of the target user, the face feature points, such as the position and shape of the eyes, nose, mouth, etc., can be extracted from the face image by using the image generation model. In another implementation, the face detection algorithm can also be used to extract the face feature points, such as the position and shape of the eyes, nose, mouth, etc., from the face image.

[0061] The face mask of the image generation template image in the embodiment of the present application can be a mask image generated based on the facial feature points of the template image. The mask image is used to identify the area in the template image that needs to be retained or modified. In the process of generating the target image, the mask image is fused with the facial image of the target user. Through the control of the mask image, it can be ensured that the specific area of ​​the template image (such as hairstyle, clothing, etc.) remains unchanged in the generated target image, while allowing the image generation model to adjust the facial details to match the facial image of the target user.

[0062] In one embodiment, the image generation model includes a feature conversion module and a feature fusion module; through the image generation model, the face mask features of the image generation template image and the face features of the target user are subjected to feature fusion processing to obtain a target image of the target user, including: Through the feature conversion module, the facial features of the target user are converted to obtain the converted facial features; Through the feature fusion module, the converted facial features and the facial attribute features of the target user are fused to obtain fused facial features; Through the image generation model, the target image of the target user is generated according to the fused facial features and the facial mask features of the image generation template image.

[0063] In one implementation, the image generation model may be composed of a feature conversion module, a feature fusion module, and an image generation module. In one example, the feature conversion module may be an Adapter module, which may perform mapping conversion on the extracted facial feature embeddings. The feature fusion module may be an Adapter fusion module.

[0064] The facial attribute features may include gender, hairstyle, age group, expression, etc. The facial attribute features may be extracted through an image generation model, or may be obtained by using an ImageCaption (image description generation) label extraction method, such as the BLIP2 model, and the extracted facial attribute features are corrected based on the feature keyword K1 obtained above. Specifically, the extracted facial attribute features are compared with the feature keyword K1, and the facial attribute features with large differences are replaced with the feature keyword K1, thereby correcting the facial attribute feature extraction results.

[0065] The feature fusion module performs feature fusion processing on the converted facial features and the facial attribute features of the target user to obtain fused facial features. The fused facial features can be used as facial image prompt information to guide image generation.

[0066] Figure 4 The following is a schematic diagram of the composition of the image generation model provided in the embodiment of the present application. The basic backbone structure of the image generation model adopts Stable Diffusion (SD, diffusion model). In order to ensure the facial fidelity of the generated avatar and meet the personalized preferences of users, the image generation model adds a facial feature adapter module and an adapter fusion module. Figure 4As shown in the figure, the working principle of the image generation model is as follows: in the process of image generation and denoising, the self-Attention (SA, self-attention mechanism) module can model the intrinsic dependence of its input information, and ensure the fidelity of the generated image by embedding the feature information of the target user's face image (target image reference image) in the SA module at different stages of denoising. First, in Step t1 of denoising, the face attribute features are embedded through the Unet (convolutional neural network architecture) down-sampling SA module. By embedding the face attribute features in the low-level SA module, the structural constraints on the generated image are realized. In Step t2 of denoising, the Adapter fusion features are embedded through the Unet up-sampling SA module, and the generated image is more consistent with the face details of the reference image by embedding the fused face features in the high-level SA module. The values ​​of t1 and t2 can control the feature embedding strength. The stronger the embedding is in the early stage of the denoising process, the greater the control strength. In addition, in order to control the style consistency effect of the generated image, the face mask features of the template image are also embedded in the last upsampling SA module. Through the mask assistance, the SA module focuses more on the dependency learning of the face area and does not interfere too much with the overall style of the generated image.

[0067] Specifically, in the downsampling stage of Unet, the denoising step t1 starts to embed facial attribute features, and the calculation method of the downsampling SA module is shown in formula (8).

[0068] ; (8) In formula (8), Qfb is the facial attribute feature, and Qm, Km, and Vm are the feature components of the image generation template map obtained by SD backbone structure processing.

[0069] In the upsampling stage of Unet, the denoising step t2 begins to embed the fused face features obtained by the Adapter fusion module. The calculation method of the upsampling SA module is shown in formula (9).

[0070] ; (9) In formula (9), Qfc is the fused face feature, and Qm, Km, and Vm are the feature components of the image generation template obtained by processing the SD backbone structure.

[0071] In the last SA module of Unet upsampling, the face mask features of the template image are embedded in the image, and the calculation method is shown in formula (10). (10) In formula (10), M is the face mask feature of the image template image. M is a binary matrix. The position value of the face area of ​​the template image is 255, and the position value of the non-face area is 0.

[0072] In the embodiment of the present application, the face mask features of the image template image can be obtained through an image generation model, and can also be obtained through a preset segmentation model.

[0073] Figure 5 This is a schematic diagram of the architecture of the Adapter module and the Adapter fusion module provided in the embodiment of the present application. Figure 5 As shown in the figure, the Adapter module includes a forward fully connected layer, a fully connected downsampling layer, a nonlinear activation layer, and a fully connected upsampling layer. The Adapter module performs feature conversion on the input facial feature embedding. The Adapter fusion module includes a softmax activation function. The Adapter fusion module is responsible for fusing the converted facial features output by the Adapter module with the facial attribute feature embedding and outputting the fused facial features.

[0074] In one embodiment, it further includes: Through the image generation model, the characteristic keywords of the target user are determined as text prompt information of the image generation model.

[0075] In an embodiment of the present application, the basic user information can reflect the user's appearance, and the user preference information can reflect the user's preferences, emotions, etc. The feature keywords K1 and K3 corresponding to the basic user information and the user preference information are input into the image generation model as text prompt information, and the generated image can be adjusted to generate an image that better suits the user's preferences.

[0076] In addition, for each type of template image, a set of customized text prompt information can be obtained through Prompt engineering debugging, and the text prompt information of the template image and the feature keywords K1 and K3 can be combined and input into the image generation model to generate a customized image.

[0077] In one embodiment, it further includes: Acquire a first sample image generation template image and a first sample image reference image, and a first sample image corresponding to the first sample image generation template image and the first sample image reference image; Determine, by means of a pre-built neural network, a first target image corresponding to the first sample image generation template image and the first sample image reference image according to the first sample image generation template image and the first sample image reference image; According to the first sample image, the first target image and the target loss function, the neural network is iteratively trained, and the trained neural network is determined as an image generation model.

[0078] In an embodiment of the present application, before generating a target image of a target user through an image generation model, a pre-built neural network may also be trained, and the trained neural network may be determined as an image generation model. First, a first sample image generation template image and a first sample image reference image are obtained, as well as a first sample image corresponding to the first sample image generation template image and the first sample image reference image. The first sample image may be a personalized avatar or image of a user that is pre-generated or acquired, and the first sample image generation template image may be a template image selected by the user when generating the first sample image. The first sample image reference image may be a face image uploaded by the user when generating the first sample image.

[0079] Then, the template image generated by the first sample image and the reference image of the first sample image are input into the pre-built neural network, and the corresponding first target image is generated through the neural network. Next, the loss in the neural network training process is determined based on the first target image and the first sample image using the loss function, and the parameters in the neural network are adjusted. The neural network is continuously iterated and trained until the loss reaches a preset loss threshold, the training is terminated, and the trained neural network is determined as the image generation model.

[0080] In an exemplary implementation, the pre-built neural network may include, but is not limited to, an Adapter module, an Adapter fusion module, and a self-Attention module.

[0081] It can be seen from the technical solution provided by the above embodiment of the present application that, in the embodiment of the present application, first, based on the image generation service request initiated by the target user, the user portrait of the target user is obtained; through the user style portrait prediction model, the user portrait and the preset image style template library are matched and processed to obtain the user style portrait prediction result of the target user; through the image generation model, the image generation template graph of the target user is generated according to the user style portrait prediction result and the preset image style template library; through the image generation model, the target image of the target user is generated according to the image generation template graph and the target image reference graph. It can be seen that through the embodiment of the present application, the user style portrait of the target user can be determined based on the user portrait of the target user and the preset image style template library through the user style portrait prediction model, and then the image generation template graph of the target user is generated through the image generation model, so that the target image of the target user can be generated according to the image generation template graph and the target image reference graph, which solves the problem that the related technology cannot automatically generate personalized images that meet the user's preferences and characteristics according to the user's preferences, characteristics and facial features, etc., so that the generated target image is more accurate and more in line with the preferences and characteristics of the target user, and the user experience is improved.

[0082] Figure 6This is a flow chart of another image generation method provided in an embodiment of the present application. The execution subject of the method is an application program, wherein the application program can be a tool application program such as a cloud disk or an email box, or a social media software such as WeChat or Weibo. The application program is installed on a terminal device used by a user, and the terminal device can be an electronic device such as a mobile phone, a tablet computer, a laptop computer, a PDA, or an in-vehicle electronic device. Figure 6 As shown, the method may specifically include the following steps: Step S602: In response to the target user triggering the target image setting operation button, a target image reference image upload interface is displayed to the target user; Step S604: in response to the target user triggering the target image reference image upload completion operation button, displaying a template image list to the target user; the template image list is determined based on the user style portrait of the target user and the preset style template library; the user style portrait of the target user is determined based on the user portrait of the target user and the preset style template library; Step S606: in response to the target user triggering the template image operation button, displaying the selected template image to the target user; Step S608: In response to the target user triggering a target image generation operation button, the target image is displayed to the target user; the target image is generated by an image generation model based on the template image selected by the target user and the target image reference image.

[0083] Corresponding to the image generation method provided in the above embodiment, based on the same technical concept, the embodiment of the present application also provides an image generation device, Figure 7 A schematic diagram of the module composition of an image generating device provided in an embodiment of the present application, wherein the user terminal is used to execute Figure 1-Figure 5 The image generation method described, such as Figure 7 As shown, the image generating device comprises: An acquisition module 701 is used to acquire a user portrait of a target user based on an image generation service request initiated by the target user; Prediction module 702, used to match the user portrait with a preset image style template library through a user style portrait prediction model to obtain a user style portrait prediction result of the target user; The first generation module 703 is used to generate an image generation template graph of the target user according to the user style portrait prediction result and the preset image style template library through an image generation model; The second generating module 704 is used to generate the target image of the target user through the image generation model according to the image generation template image and the target image reference image.

[0084] In a possible implementation, the prediction module 702 is specifically configured to determine the degree of match between the user portrait and each template image in the preset image style template library through the user style portrait prediction model; The user style portrait prediction model is used to determine a user style portrait prediction result of the target user according to a matching degree between the user portrait and each template image in the preset image style template library.

[0085] In a possible implementation, the prediction module 702 is further specifically configured to perform feature keyword extraction processing on the user profile of the target user through the user style portrait prediction model to obtain the feature keywords of the target user; By means of the user style portrait prediction model, the feature keywords and prompt information are encoded to obtain the user portrait text features; the prompt information includes the style category of the template image and the context information representing the style of the template image; By using the user style portrait prediction model, feature extraction processing is performed on each template image in the preset image style template library to obtain a representation feature of the template image; The user style portrait prediction model is used to determine the degree of match between the user portrait and each of the template images according to the user portrait text features and the representation features of each of the template images.

[0086] In a possible implementation, the prediction module 702 is further specifically configured to determine, through the user style portrait prediction model, each matching degree between the user portrait and each template image in the preset image style template library as a user style portrait prediction result of the target user.

[0087] In a possible implementation, the first generating module 703 is specifically configured to obtain a weight of each template image in the preset image style template library; The image generation model is used to perform weighted processing on the user style portrait prediction result and each template image in the preset image style template library to obtain an image generation template graph of the target user.

[0088] In a possible implementation, the first generation module 703 is further specifically configured to perform weighted processing on the user style portrait prediction result and the weight of each template image through the image generation model to obtain a recommendation weight of each template image; According to the recommendation weight of each template image, sorting the template images to generate a template image list; In response to the target user triggering an operation of selecting a template image in the template image list, the template image selected by the target user is acquired, and the template image is determined as an image generation template image of the target user.

[0089] In a possible implementation, the target image reference image includes a facial image of the target user; the second generation module 704 is specifically configured to perform feature extraction processing on the facial image of the target user through the image generation model to obtain facial features of the target user; The image generation model is used to perform feature fusion processing on the face mask features of the image generation template image and the face features of the target user to obtain a target image of the target user.

[0090] In a possible implementation, the image generation model includes a feature conversion module and a feature fusion module; the second generation module 704 is further specifically configured to convert the facial features of the target user through the feature conversion module to obtain converted facial features; By means of the feature fusion module, the converted facial features and the facial attribute features of the target user are subjected to feature fusion processing to obtain fused facial features; The target image of the target user is generated through the image generation model according to the fused facial features and the facial mask features of the image generation template image.

[0091] In a possible implementation, the image generation device further includes a determination module, which is used to determine the characteristic keywords of the target user as text prompt information of the image generation model through the image generation model.

[0092] In a possible implementation, the image generating device further includes a training module, which is used to obtain a first sample image generating template image and a first sample image reference image, and a first sample image corresponding to the first sample image generating template image and the first sample image reference image; Determine, by a pre-built neural network, a first target image corresponding to the first sample image generation template image and the first sample image reference image according to the first sample image generation template image and the first sample image reference image; The neural network is iteratively trained according to the first sample image, the first target image and the target loss function, and the trained neural network is determined as the image generation model.

[0093] It can be seen from the technical solution provided by the above embodiment of the present application that, in the embodiment of the present application, first, based on the image generation service request initiated by the target user, the user portrait of the target user is obtained; through the user style portrait prediction model, the user portrait and the preset image style template library are matched and processed to obtain the user style portrait prediction result of the target user; through the image generation model, the image generation template graph of the target user is generated according to the user style portrait prediction result and the preset image style template library; through the image generation model, the target image of the target user is generated according to the image generation template graph and the target image reference graph. It can be seen that through the embodiment of the present application, the user style portrait of the target user can be determined based on the user portrait of the target user and the preset image style template library through the user style portrait prediction model, and then the image generation template graph of the target user is generated through the image generation model, so that the target image of the target user can be generated according to the image generation template graph and the target image reference graph, which solves the problem that the related technology cannot automatically generate personalized images that meet the user's preferences and characteristics according to the user's preferences, characteristics and facial features, etc., so that the generated target image is more accurate and more in line with the preferences and characteristics of the target user, and the user experience is improved.

[0094] It should be noted that the image generating device provided in the embodiment of the present application and the image generating method provided in the embodiment of the present application are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the aforementioned image generating method, and the repeated parts will not be repeated.

[0095] Corresponding to the image generation method provided in the above embodiment, based on the same technical concept, the embodiment of the present application further provides an electronic device, which is used to execute the above image generation method. Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 801 and memory 802, and the memory 802 may store one or more storage applications or data. Among them, the memory 802 may be a temporary storage or a permanent storage. The application stored in the memory 802 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 801 may be configured to communicate with the memory 802 to execute a series of computer executable instructions in the memory 802 on the electronic device. The electronic device may also include one or more power supplies 803, one or more wired or wireless network interfaces 804, one or more input and output interfaces 805, and one or more keyboards 806.

[0096] Specifically in this embodiment, the electronic device includes a processor, a communication interface, a memory and a communication bus; wherein the processor, the communication interface and the memory communicate with each other through the bus; the memory is used to store a computer program; the processor is used to execute the program stored in the memory to implement the following method steps: Based on the image generation service request initiated by the target user, a user portrait of the target user is obtained; The user style portrait prediction model is used to match the user portrait with a preset image style template library to obtain a user style portrait prediction result of the target user; Generate an image generation template graph of the target user according to the user style portrait prediction result and the preset image style template library through the image generation model; The target image of the target user is generated by the image generation model according to the image generation template image and the target image reference image.

[0097] It can be seen from the technical solution provided by the above embodiment of the present application that, in the embodiment of the present application, first, based on the image generation service request initiated by the target user, the user portrait of the target user is obtained; through the user style portrait prediction model, the user portrait and the preset image style template library are matched and processed to obtain the user style portrait prediction result of the target user; through the image generation model, the image generation template graph of the target user is generated according to the user style portrait prediction result and the preset image style template library; through the image generation model, the target image of the target user is generated according to the image generation template graph and the target image reference graph. It can be seen that through the embodiment of the present application, the user style portrait of the target user can be determined based on the user portrait of the target user and the preset image style template library through the user style portrait prediction model, and then the image generation template graph of the target user is generated through the image generation model, so that the target image of the target user can be generated according to the image generation template graph and the target image reference graph, which solves the problem that the related technology cannot automatically generate personalized images that meet the user's preferences and characteristics according to the user's preferences, characteristics and facial features, etc., so that the generated target image is more accurate and more in line with the preferences and characteristics of the target user, and the user experience is improved.

[0098] The present application also provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the following method steps are implemented: Based on the image generation service request initiated by the target user, a user portrait of the target user is obtained; The user style portrait prediction model is used to match the user portrait with a preset image style template library to obtain a user style portrait prediction result of the target user; Generate an image generation template graph of the target user according to the user style portrait prediction result and the preset image style template library through the image generation model; The target image of the target user is generated by the image generation model according to the image generation template image and the target image reference image.

[0099] It can be seen from the technical solution provided by the above embodiment of the present application that, in the embodiment of the present application, first, based on the image generation service request initiated by the target user, the user portrait of the target user is obtained; through the user style portrait prediction model, the user portrait and the preset image style template library are matched and processed to obtain the user style portrait prediction result of the target user; through the image generation model, the image generation template graph of the target user is generated according to the user style portrait prediction result and the preset image style template library; through the image generation model, the target image of the target user is generated according to the image generation template graph and the target image reference graph. It can be seen that through the embodiment of the present application, the user style portrait of the target user can be determined based on the user portrait of the target user and the preset image style template library through the user style portrait prediction model, and then the image generation template graph of the target user is generated through the image generation model, so that the target image of the target user can be generated according to the image generation template graph and the target image reference graph, which solves the problem that the related technology cannot automatically generate personalized images that meet the user's preferences and characteristics according to the user's preferences, characteristics and facial features, etc., so that the generated target image is more accurate and more in line with the preferences and characteristics of the target user, and the user experience is improved.

[0100] The embodiment of the present application also provides a computer program product. When the computer program is executed by a processor, each process of the above-mentioned image generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0101] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0102] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0105] In a typical configuration, an electronic device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0106] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0107] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0108] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0109] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0110] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image generation method, characterized in that: include: Based on the image generation service request initiated by the target user, a user portrait of the target user is obtained; The user style portrait prediction model is used to match the user portrait with a preset image style template library to obtain a user style portrait prediction result of the target user; Generate an image generation template graph of the target user according to the user style portrait prediction result and the preset image style template library through the image generation model; The target image of the target user is generated by the image generation model according to the image generation template image and the target image reference image.

2. The method according to claim 1, characterized in that The user style portrait prediction model is used to match the user portrait with a preset image style template library to obtain a user style portrait prediction result of the target user, including: Determining the matching degree between the user portrait and each template image in the preset image style template library through the user style portrait prediction model; The user style portrait prediction model is used to determine a user style portrait prediction result of the target user according to a matching degree between the user portrait and each template image in the preset image style template library.

3. The method according to claim 2, characterized in that Determining the matching degree between the user portrait and each template image in the preset image style template library by using the user style portrait prediction model includes: By using the user style portrait prediction model, feature keyword extraction processing is performed on the user portrait of the target user to obtain the feature keywords of the target user; By means of the user style portrait prediction model, the feature keywords and prompt information are encoded to obtain the user portrait text features; the prompt information includes the style category of the template image and the context information representing the style of the template image; By using the user style portrait prediction model, feature extraction processing is performed on each template image in the preset image style template library to obtain a representation feature of the template image; The user style portrait prediction model is used to determine the degree of match between the user portrait and each of the template images according to the user portrait text features and the representation features of each of the template images.

4. The method according to claim 2, characterized in that: The method of determining the user style portrait prediction result of the target user according to the matching degree between the user portrait and each template image in the preset image style template library by using the user style portrait prediction model includes: Through the user style portrait prediction model, each matching degree between the user portrait and each template image in the preset image style template library is determined as a user style portrait prediction result of the target user.

5. The method according to claim 4, characterized in that The method of generating an image generation template graph of the target user through the image generation model according to the user style portrait prediction result and the preset image style template library includes: Obtaining the weight of each template image in the preset image style template library; The image generation model is used to perform weighted processing on the user style portrait prediction result and each template image in the preset image style template library to obtain an image generation template graph of the target user.

6. The method according to claim 5, characterized in that The method of performing weighted processing on the user style portrait prediction result and each template image in the preset image style template library through the image generation model to obtain the image generation template graph of the target user includes: By using the image generation model, weighted processing is performed on the user style portrait prediction result and the weight of each template image to obtain a recommendation weight of each template image; According to the recommendation weight of each template image, sorting the template images to generate a template image list; In response to the target user triggering an operation of selecting a template image in the template image list, the template image selected by the target user is acquired, and the template image is determined as an image generation template image of the target user.

7. The method according to claim 1, characterized in that The target image reference graph includes a face image of the target user; and the generating a target image of the target user by using the image generation model and generating a template graph and a target image reference graph according to the image, comprising: Performing feature extraction processing on the face image of the target user through the image generation model to obtain the face features of the target user; The image generation model is used to perform feature fusion processing on the face mask features of the image generation template image and the face features of the target user to obtain a target image of the target user.

8. The method according to claim 1, characterized in that The image generation model includes a feature conversion module and a feature fusion module; the image generation model is used to perform feature fusion processing on the face mask features of the image generation template image and the face features of the target user to obtain the target image of the target user, including: The facial features of the target user are converted by the feature conversion module to obtain converted facial features; By means of the feature fusion module, the converted facial features and the facial attribute features of the target user are subjected to feature fusion processing to obtain fused facial features; The target image of the target user is generated through the image generation model according to the fused facial features and the facial mask features of the image generation template image.

9. The method according to claim 8, characterized in that Also includes: Through the image generation model, the characteristic keywords of the target user are determined as text prompt information of the image generation model.

10. The method according to claim 9, characterized in that Also includes: Acquire a first sample image generation template image and a first sample image reference image, and a first sample image corresponding to the first sample image generation template image and the first sample image reference image; Determine, by a pre-built neural network, a first target image corresponding to the first sample image generation template image and the first sample image reference image according to the first sample image generation template image and the first sample image reference image; The neural network is iteratively trained according to the first sample image, the first target image and the target loss function, and the trained neural network is determined as the image generation model.

Images