Image generation method and device

Abstract

The invention relates to an image generation method and device. The method comprises the following steps: providing an image selection interface; the image selection interface is used for acquiring a selected to-be-processed image; displaying a target image of the first object and / or image creation images of at least one group of theme scenes; the target image image of the first object is an image generated based on an individual image model of the first object, the image creation image of each group of theme scenes is an image generated based on the combination of a corresponding theme template model and the individual image model, and the individual image model is a model obtained based on to-be-processed image training. According to the embodiment of the invention, a user can quickly obtain the target image and the image creation image thereof, and the stability and quality of a real-person photo effect can be improved.

Description

Technical Field

[0001] The present disclosure relates to fields such as image processing and artificial intelligence, and in particular to an image generation method and device thereof. Background Art

[0002] In the related technology, AI (Artificial Intelligence) drawing tools can be used to generate corresponding images based on prompt information. For example, Midjourney, as one of the most popular AI image generators today, is an AI drawing tool based on a deep learning model such as Generative Adversarial Networks (GAN), and its core technology is text-to-image generation. Users need to enter text descriptions, and Midjourney can convert these semantic information into visual elements and generate various types of images. At the same time, the Prompt can be continuously modified and regenerated according to the generated results until it is adjusted to meet the expected effect.

[0003] However, the use threshold of AI drawing tools (such as Midjourney) in related technologies is relatively high. For example, Midjourney requires users to have certain qualities such as painting skills and color matching, and the controllability of the generated effect is not high. It is necessary to adjust the prompt multiple times to improve the effect, which makes it difficult to quickly and accurately obtain the expected image. In addition, AI drawing tools in related technologies are prone to generate bizarre effects with deviations from authenticity in the creation of portrait effects and photographic styles. Summary of the invention

[0004] The present disclosure provides an image generating method and a device thereof.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image generating method, comprising:

[0006] Providing an image selection interface; the image selection interface is used to obtain the selected image to be processed;

[0007] Display the target image of the first object and / or at least one group of image creation images of the theme scene; the target image of the first object is an image generated based on the individual image model of the first object, and each group of image creation images of the theme scene is an image generated based on the corresponding theme template model combined with the individual image model, and the individual image model is a model trained based on the image to be processed.

[0008] In combination with some embodiments of the first aspect, in some embodiments, the method also includes: performing face detection on the acquired image to be processed, and filtering out the image of the first object from the image to be processed; performing image processing on the image of the first object to obtain an image sample of the individual image model; and generating the individual image model of the first object based on the image sample.

[0009] In combination with some embodiments of the first aspect, in some embodiments, generating the individual image model of the first object based on the image sample includes: reasoning on the image sample to obtain description information of the image sample; adding reference noise to the image sample to obtain a first mixed noise image; inputting the description information and the first mixed noise image into a preset extended model to obtain predicted noise output by the extended model; adjusting the network parameters of the extended model according to the loss function of the predicted noise and the reference noise until the model training end condition is met, and determining the trained extended model as the individual image model of the first object.

[0010] In combination with some embodiments of the first aspect, in some embodiments, the image processing includes at least one of the following: face ratio detection processing, used to filter images whose face ratio is less than or equal to a first threshold; quality detection processing, used to filter images whose blur value is less than or equal to a second threshold; occlusion detection processing, used to filter images with occluded faces; angle detection processing, used to filter images whose face posture angles meet the first condition; human body detection processing, used to filter images containing multiple human bodies.

[0011] In combination with some embodiments of the first aspect, in some embodiments, the method also includes: obtaining a reference image from the image of the first object; performing inference on the reference image to obtain description information of the reference image; adding reference noise to the reference image to obtain a second mixed noise image; inputting the description information of the reference image and the second mixed noise image into the individual image model of the first object to obtain the predicted noise output by the individual image model; and generating a target image image of the first object based on the predicted noise and the second mixed noise image.

[0012] In combination with some embodiments of the first aspect, in some embodiments, the method also includes: fusing at least one preset theme template model with the individual image model to obtain at least one fusion model; and generating an image creation image of the at least one group of theme scenes based on the image of the first object and the at least one fusion model.

[0013] In combination with some embodiments of the first aspect, in some embodiments, the method also includes: providing a theme template model selection interface; the theme template model selection interface is used to obtain a selected theme template model; and based on the selected theme template model, generating an image creation image corresponding to the theme scene.

[0014] In combination with some embodiments of the first aspect, in some embodiments, the method further includes: providing an input interface; the input interface is used to obtain input text description information; and generating a corresponding image based on the text description information and the individual image model of the first object.

[0015] According to a second aspect of an embodiment of the present disclosure, there is provided an image generating device, including:

[0016] A first providing module is used to provide an image selection interface; the image selection interface is used to obtain a selected image to be processed;

[0017] A display module is used to display a target image of a first object and / or at least one group of image creation images of a theme scene; the target image of the first object is an image generated based on an individual image model of the first object, and each group of image creation images of the theme scene is an image generated based on a corresponding theme template model combined with the individual image model, and the individual image model is a model trained based on the image to be processed.

[0018] In combination with some embodiments of the second aspect, in some embodiments, the device may also include: a face detection module, used to perform face detection on the acquired image to be processed, and filter out the image of the first object from the image to be processed; an image processing module, used to perform image processing on the image of the first object to obtain an image sample of the individual image model; and a first generation module, used to generate the individual image model of the first object based on the image sample.

[0019] In combination with some embodiments of the second aspect, in some embodiments, the first generation module is specifically used to: perform inference on image samples to obtain description information of the image samples; add reference noise to the image samples to obtain a first mixed noise image; input the description information and the first mixed noise image into a preset extended model to obtain the predicted noise output by the extended model; adjust the network parameters of the extended model according to the loss function of the predicted noise and the reference noise until the model training end conditions are met, and determine the trained extended model as the individual image model of the first object.

[0020] In combination with some embodiments of the second aspect, in some embodiments, image processing includes at least one of the following: face ratio detection processing, used to filter images whose face ratio is less than or equal to a first threshold; quality detection processing, used to filter images whose blur value is less than or equal to a second threshold; occlusion detection processing, used to filter images with occluded faces; angle detection processing, used to filter images whose face posture angles meet a first condition; and human body detection processing, used to filter images containing multiple human bodies.

[0021] In combination with some embodiments of the second aspect, in some embodiments, the device may also include: a first acquisition module, used to acquire a reference image from the image of the first object; an inference module, used to infer the reference image to obtain description information of the reference image; an addition module, used to add reference noise to the reference image to obtain a second mixed noise image; a second acquisition module, used to input the description information of the reference image and the second mixed noise image into the individual image model of the first object to obtain the predicted noise output by the individual image model; a second generation module, used to generate a target image image of the first object based on the predicted noise and the second mixed noise image.

[0022] In combination with some embodiments of the second aspect, in some embodiments, the device may also include: a fusion module, used to fuse at least one preset theme template model with the individual image model to obtain at least one fusion model; a third generation module, used to generate at least one group of image creation images of the theme scene based on the image of the first object and at least one fusion model.

[0023] In combination with some embodiments of the second aspect, in some embodiments, the device may also include: a second providing module, used to provide a theme template model selection interface; the theme template model selection interface is used to obtain the selected theme template model; and a fourth generating module, used to generate an image creation image corresponding to the theme scene based on the selected theme template model.

[0024] In combination with some embodiments of the second aspect, in some embodiments, the device may also include: a third providing module, used to provide an input interface; the input interface is used to obtain input text description information; a fifth generating module, used to generate a corresponding image based on the text description information and the individual image model of the first object.

[0025] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0026] processor;

[0027] A memory for storing processor-executable instructions; wherein the instructions are executed by the processor so that the processor can execute the method described in the first aspect above.

[0028] According to a fourth aspect of an embodiment of the present disclosure, a readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in the first aspect above.

[0029] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor of an electronic device, the method described in the first aspect is implemented.

[0030] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0031] By providing an image selection interface, the user can provide the image selection interface to select an image to be processed for image creation, so that the electronic device can generate a personal image model of the first object based on the selected image to be processed, and can generate a target image image of the first object based on the personal image model. It can also generate an image creation image of a theme scene based on the personal image model of the first object combined with a theme template model. After obtaining the target image of the first object and the image creation image of the theme scene, the generated target image of the first object and / or the image creation image of the theme scene can be displayed. In the whole process, the user does not need to perform too much operation, and the operation is simple for the user, and the threshold for use is low, so that the user can quickly obtain the target image image and its image creation image. In addition, the embodiment of the present disclosure can improve the stability and quality of the real-life photo effect by using a personalized individual image model to generate the corresponding target image image and image creation image.

[0032] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0034] Figure 1 The figure is a flowchart of an image generating method according to an exemplary embodiment.

[0035] Figure 2A is a diagram showing an example of an image selection interface according to an exemplary embodiment.

[0036] Figure 2B The figure is an example diagram of a display interface according to an exemplary embodiment.

[0037] Figure 3The figure is a flowchart of an image generating method according to an exemplary embodiment.

[0038] Figure 4 The figure is a block diagram of an image generating device according to an exemplary embodiment.

[0039] Figure 5 The figure is a block diagram of an image generating device according to an exemplary embodiment.

[0040] Figure 6 The figure is a block diagram of an image generating device according to an exemplary embodiment.

[0041] Figure 7 The figure is a block diagram of an image generating device according to an exemplary embodiment.

[0042] Figure 8 The figure is a block diagram of an image generating device according to an exemplary embodiment.

[0043] Fig. 9 The figure is a block diagram of an image generating device according to an exemplary embodiment.

[0044] Fig.10 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0045] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0046] It should be noted that, in the description of the present disclosure, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in this article is merely a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0047] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present disclosure. The singular forms "a", "an" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0048] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the disclosed embodiments, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the disclosed embodiments, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at" or "when" or "in response to determination".

[0049] It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0050] Figure 1 1 is a flow chart of an image generation method according to an exemplary embodiment. It should be noted that the image generation method is used in an electronic device, and as an example, the electronic device may be a terminal (such as a mobile phone, a tablet computer, a personal digital assistant, a PC, and other terminal devices with various hardware systems).

[0051] like Figure 1 As shown, the method may include but is not limited to the following steps.

[0052] In step 101, an image selection interface is provided; the image selection interface is used to obtain a selected image to be processed.

[0053] Exemplarily, the disclosed embodiments are used in an electronic device, which provides an application with an image creation function for a user. The application can provide an image selection interface for the user. The user can select an image through the image selection interface, so that the electronic device can obtain the selected image to be processed through the image selection interface.

[0054] In some embodiments, the selected image to be processed may be a person image, which may be an image that has been intelligently aggregated based on the user's photo album. Figure 2A As shown, the electronic device may provide a target image creation portal, and when the portal is triggered (e.g., clicked), an image selection interface is entered, that is, the image selection interface may be displayed on the electronic device, and the image selection interface may have a character (e.g., Zhang San, Li Si, etc.) image selection control, and may also have a "manual image selection" control and the like. The user determines the selected image to be processed by triggering the character image selection control. For example, if the user selects the image of Zhang San, the electronic device may determine the character image of Zhang San as the selected image to be processed.

[0055] In some embodiments, the above-mentioned character image may also be an image input by the user. Exemplarily, the above-mentioned character image may be an image selected by the user from the photo album through the "manual image selection" control in the image selection interface. It should be noted that the image to be processed selected in the embodiment of the present disclosure may be a character image in the user's photo album, an image input by the user, or an image obtained by other means, which is not limited in the present disclosure and will not be elaborated.

[0056] In some embodiments, the image selection interface may include prompt information, and the prompt information may guide the user to create an image, thereby facilitating the user's use.

[0057] In step 102, a target image of a first object and / or an image creation image of at least one group of subject scenes are displayed.

[0058] In some embodiments, a target image of the first object may be displayed.

[0059] In some embodiments, at least one set of character creation images of a themed scene may be displayed.

[0060] In some embodiments, a target image of a first object and image creation images of at least one group of subject scenes may be displayed.

[0061] In some embodiments, the target image of the first object may be an image generated based on an individual image model of the first object. The individual image model of the first object may be a model trained based on image samples, and the image samples may be images obtained by performing face detection and image processing on the selected image to be processed. The individual image model of the first object may be used to predict noise.

[0062] In a possible implementation, an image of the first object can be selected from the above-selected images to be processed, a corresponding mixed noise image can be generated based on the image of the first object, and the predicted noise of the mixed noise image can be obtained based on the mixed noise image and the individual image model of the first object. The target image of the first object can be obtained based on the mixed noise image and the predicted noise. Exemplarily, when the target image of the first object is obtained, the target image of the first object can be displayed, for example, as shown in FIG. Figure 2B As shown, the electronic device provides a display interface, which can display the generated target image of the first object in a specific area. For example, the generated target image of the first object can be displayed in a specific area. Figure 2B In the first area A in .

[0063] In some embodiments, the image creation images of each group of theme scenes in the at least one group of theme scenes may be images generated based on the corresponding theme template model combined with the individual image model. Each group of theme scene image creation images may include one or more image creation images of the same theme scene.

[0064] In a possible implementation, at least one preset theme template model may be fused with the individual image model of the first object, and at least one group of image creation images of the theme scene may be generated based on the fused models.

[0065] Exemplarily, taking the example that at least one theme template model includes theme template model 1 and theme template model 2, theme template model 1 can be fused with the individual image model of the first object, and one or more image creation images of the corresponding theme scene (such as theme scene A) are generated based on the fused model, and the one or more image creation images are images of the same theme scene. The theme template model 2 is fused with the individual image model of the first object, and one or more image creation images of the corresponding theme scene (such as theme scene B) are generated based on the fused model. Exemplarily, when at least one group of image creation images of theme scenes are obtained, at least one group of image creation images of theme scenes can be displayed. For example, taking the example that the generated at least one group of image creation images of theme scenes includes image creation images of theme scene A and image creation images of theme scene B, as Figure 2B As shown, the electronic device provides a display interface, which can display the generated target image of the first object in a specific area. For example, the generated image creation image of the theme scene A (such as 4 images of the same theme scene) can be displayed in the following area: Figure 2B In the second area B in the image creation image of the theme scene B (such as 4 images of the same theme scene) are displayed on the Figure 2B In the third area C.

[0066] In some embodiments, the individual image model of the first object may be a model trained based on the image to be processed. Exemplarily, face detection and image processing may be performed on the selected image to be processed to obtain an image sample in which the face belongs to the first object, and the image sample may be used for model training to generate the individual image model of the first object.

[0067] In the above embodiment, by providing an image selection interface, the user can provide the image selection interface to select the image to be processed for image creation, so that the electronic device can generate a personal image model of the first object based on the selected image to be processed, and the target image image of the first object can be generated based on the personal image model. It can also generate an image creation image of the theme scene based on the personal image model of the first object combined with the theme template model. After obtaining the target image of the first object and the image creation image of the theme scene, the generated target image of the first object and / or the image creation image of the theme scene can be displayed. In the whole process, the user does not need to perform too much operation, the operation is simple for the user, and the threshold for use is low, so that the user can quickly obtain the target image image and its image creation image. In addition, the embodiment of the present disclosure can improve the stability and quality of the real-life photo effect by using a personalized individual image model to generate the corresponding target image image and image creation image.

[0068] Figure 3 FIG. 1 is a flow chart of an image generation method according to an exemplary embodiment. Figure 3 As shown, the method may include but is not limited to the following steps.

[0069] In step 301, an image selection interface is provided; the image selection interface is used to obtain a selected image to be processed.

[0070] In the embodiments of the present disclosure, the implementation of step 301 may be implemented in any of the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0071] In step 302, face detection is performed on the acquired image to be processed, and an image of the first object is screened out from the image to be processed.

[0072] In some embodiments, a face detection algorithm may be used to perform face detection on the acquired image to be processed, and images without faces or with multiple faces may be filtered to select the image of the first object from the image to be processed. The first object may be a user using the image creation function or another user, that is, the user may generate the user's own target image and image creation images under the theme scene by using the image generation method provided in the embodiment of the present disclosure, and may also generate target image images and image creation images under the theme scene of other users, which are not specifically limited here.

[0073] In step 303, image processing is performed on the image of the first object to obtain an image sample of the individual image model.

[0074] Optionally, the image of the first object may be processed to obtain a high-quality image for training the individual image model of the first object.

[0075] In some embodiments, the above-mentioned image processing may include but is not limited to at least one of the following: face ratio detection processing; quality detection processing; occlusion detection processing; angle detection processing; human body detection processing. In this embodiment, the above-mentioned image processing may include face ratio detection processing, and face ratio detection processing may be performed on the image of the first object. In this embodiment, the above-mentioned image processing may include quality detection processing, and quality detection processing may be performed on the image of the first object. In this embodiment, the above-mentioned image processing may include occlusion detection processing, and occlusion detection processing may be performed on the image of the first object. In this embodiment, the above-mentioned image processing may include angle detection processing, and angle detection processing may be performed on the image of the first object. In this embodiment, the above-mentioned image processing may include human body detection processing, and human body detection processing may be performed on the image of the first object. It should be noted that the above-mentioned embodiments are not exhaustive, but are only illustrative of some embodiments, and the above-mentioned embodiments may be implemented separately or in combination. The above-mentioned embodiments are only for illustration and are not for specific limitation of the protection scope of the embodiments of the present disclosure.

[0076] In some embodiments, the above-mentioned face ratio detection process is used to filter images whose face ratio is less than or equal to a first threshold. Exemplarily, the face part detected by the face (such as the face part in the image of the first object) can be filtered by ratio, for example, images whose face ratio is less than or equal to a first threshold (such as 50%) can be filtered.

[0077] In some embodiments, the quality detection process is used to filter images whose blur value is less than or equal to the second threshold. Exemplarily, the Laplace operator can be used to detect image blur, and images whose blur value is less than or equal to the second threshold can be filtered. Optionally, other image blur detection algorithms can be used to calculate the blur of the image, which is not specifically limited.

[0078] In some embodiments, the occlusion detection process is used to filter images with occlusion on the face. Exemplarily, an occlusion detection algorithm may be used to filter images with occlusion on the face.

[0079] Among them, in some embodiments, the above-mentioned angle detection processing is used to filter images whose facial posture angles meet the first condition. Exemplarily, a facial posture estimation algorithm can be used to estimate the facial posture of the facial part detected by the face (such as the facial part in the image of the first object), determine the posture angle of the face in the image, such as at least one of the pitch angle, roll angle, and yaw angle, and filter images whose facial posture angles are greater than or equal to a certain threshold. For example, taking the pitch angle as an example, images whose pitch angles are greater than or equal to a threshold (such as 45 degrees) can be filtered.

[0080] In some embodiments, the human body detection process is used to filter images containing multiple human bodies. Exemplarily, individual detection can be performed on the face part detected by the face (such as the face part in the image of the first object) to filter the images containing multiple human bodies.

[0081] In step 304, an individual image model of the first object is generated based on the image sample.

[0082] Optionally, in some embodiments, the image samples may be processed, such as by cropping out a partial area of ​​character features that the model needs to focus on learning, so as to improve the learning of character features, and then using the processed image samples for model training to generate an individual image model of the first object.

[0083] In some embodiments, reference noise can be added to an image sample to obtain a first mixed noise image, the description information of the image sample and the first mixed noise image are input into a preset extended model to obtain the predicted noise output by the extended model, the extended model is trained based on the reference noise and the predicted noise, and the trained extended model is determined as the individual image model of the first object.

[0084] In a possible implementation, inference is performed on image samples to obtain description information of the image samples; reference noise is added to the image samples to obtain a first mixed noise image; the description information and the first mixed noise image are input into a preset extended model to obtain predicted noise output by the extended model; based on a loss function of the predicted noise and the reference noise, the network parameters of the extended model are adjusted until the model training end conditions are met, and the trained extended model is determined as the individual image model of the first object.

[0085] Exemplarily, an image description model may be used to infer an image sample to obtain description information of the image sample. The image description model may be pre-trained. As an example, the image description model may be trained using an attention model, or may be trained using other models, which are not specifically limited here. The input of the image description model may be an image, and the output of the image description model is the description information of the image, which may be a text sentence that can describe the image scene.

[0086] Exemplarily, after adding reference noise to an image sample, a first mixed noise image is obtained, and the description information of the image sample and the first mixed noise image can be input into the extended model for noise prediction to obtain the predicted noise output by the extended model. A preset loss function is used to calculate the loss function value between the predicted noise and the reference noise, and the network parameters of the extended model are adjusted according to the loss function value until the model training end condition is met, and the trained extended model is determined as the individual image model of the first object.

[0087] In step 305, a target image of the first object is generated based on the individual image model of the first object.

[0088] In some embodiments, an image of the first object can be selected from the above-mentioned selected images to be processed, a corresponding mixed noise image can be generated according to the image of the first object, and the predicted noise of the mixed noise image can be obtained according to the mixed noise image and the individual image model of the first object. The target image of the first object can be obtained according to the mixed noise image and the predicted noise.

[0089] In a possible implementation, a reference image can be obtained from an image of a first object. Inference is performed on the reference image to obtain description information of the reference image. Reference noise is added to the reference image to obtain a second mixed noise image. The description information of the reference image and the second mixed noise image are input into an individual image model of the first object to obtain predicted noise output by the individual image model. A target image of the first object is generated based on the predicted noise and the second mixed noise image.

[0090] Exemplarily, a pre-trained image description model may be used to infer the reference image to obtain description information of the reference image. The description of the image description model can refer to the above description of the image description model, which will not be repeated here.

[0091] Exemplarily, after the description information of the reference image and the second mixed noise image are input into the individual image model of the first object and the predicted noise output by the individual image model is obtained, the second mixed noise image can be differentially processed from the predicted noise, and the image obtained after the differential processing can be determined as the target image image of the first object.

[0092] In step 306, at least one set of image creation images of the theme scene is generated based on at least one preset theme template model and the individual image model of the first object.

[0093] Optionally, at least one theme template model may be fused with the individual image model of the first object, and an image creation image of the corresponding theme scene may be generated based on each fused model.

[0094] In some embodiments, at least one theme template model is fused with the individual image model of the first object to obtain at least one fusion model. At least one set of theme scene image creation images is generated based on the image of the first object and the at least one fusion model.

[0095] In one possible implementation, for each theme template model, the theme template model can be fused with the individual image model of the first object. Exemplarily, the model parameters of the theme template model can be added to the model parameters of the individual image model of the first object to achieve the fusion of the two models. Based on the image of the first object and the fusion model obtained after the fusion of the two models, an image creation image of the theme scene can be obtained.

[0096] Exemplarily, taking the theme template model 1 of theme scene A as an example, the model parameters of theme template model 1 and the model parameters of the individual image model of the first object are added to realize the fusion processing of the two models to obtain a fusion model. The image of the first object can be inferred to obtain the description information of the image. Reference noise is added to the image of the first object to obtain a mixed noise image, the description information and the mixed noise image are input into the fusion model, the predicted noise output by the fusion model is obtained, the mixed noise image and the predicted noise are differentially processed, and the image after differential processing is determined as the image creation image of theme scene A. When there are multiple images of the first object, multiple image creation images of theme scene A are generated.

[0097] In some embodiments, the image of the first object may be a target image of the first object, or may be an image of a human face belonging to the first object among the selected images to be processed.

[0098] In some embodiments, the at least one theme template model may be a portrait model under at least one different theme scene. The theme template model may be pre-trained. For example, a portrait model under different theme scenes may be trained based on a large amount of theme materials, and the theme materials may be theme materials photographed using professional equipment and / or professional means. The training logic of the theme template model is the same as the training logic of the individual image model of the first object. Exemplarily, for each theme scene, the theme material image of the theme scene may be inferred to obtain description information of the theme material image; reference noise is added to the theme material image to obtain a mixed noise image; the mixed noise image and the description information of the theme material image are input into the extended model to obtain the predicted noise output by the extended model, and the network parameters of the extended model are adjusted according to the loss function of the predicted noise and the reference noise until the model training end condition is met, and the trained extended model is used to determine the theme template model of the theme scene.

[0099] In step 307, a target image of the first object and at least one group of image creation images of a theme scene are displayed.

[0100] In some embodiments, an image display interface may be provided, and the generated target image of the first object and at least one group of image creation images of the subject scene may be displayed on the image display interface.

[0101] In order to further enhance the user experience and realize the image creation of multiple different theme scenes, multiple different theme scenes can be provided for the user to choose, so that the user can create images by selecting different theme scenes. In some embodiments, a theme template model selection interface can be provided; the theme template model selection interface is used to obtain the selected theme template model. According to the selected theme template model, an image creation image of the corresponding theme scene is generated. Exemplarily, there is a "more" control in the image display interface. After the user clicks the "more" control, the theme template model selection interface can be displayed. The theme template model selection interface includes theme template models of multiple different theme scenes for the user to choose. After the user completes the selection operation, the image creation image of the theme scene can be generated according to the selected theme template model and the image of the first object. Exemplarily, the selected theme template model is fused with the individual image model of the first object to obtain a fusion model, and the image creation image of the theme scene is generated according to the image of the first object and the fusion model.

[0102] In order to further enhance the user experience and the image creation effect, a text input interface may be provided for the user, thereby satisfying the creation requirements of the free input prompt. In some embodiments, an input interface may be provided, which is used to obtain input text description information; based on the text description information and the individual image model of the first object, a corresponding image is generated. Exemplarily, when the text description information input by the user is obtained through the input interface, the corresponding image may be generated based on the text description information and the individual image model of the first object using the text description technology.

[0103] In the above embodiment, the user does not need to perform too many operations. For the user, the operation is simple and the threshold for use is low, so that the user can quickly obtain the target image and its image creation image. In addition, the embodiment of the disclosure can improve the stability and quality of the real-life photo effect by using a personalized individual image model to generate the corresponding target image and image creation image.

[0104] It should be noted that, in some embodiments, the process of generating the individual image model of the first object may be implemented on the electronic device (such as a terminal) side or on the server side.

[0105] In summary, the present disclosure provides an image generation scheme that can quickly create photo images based on album character images. Users can easily share others, use social media avatars, wallpapers, emoticons, etc.; the present disclosure provides a fast and convenient template creation experience, and can obtain stable and high-quality real-life photo effects, and also meet the needs of free input prompts for the creation of literary images. Specifically, the present disclosure can filter out multiple (such as 20 to 30) high-quality character images based on the smart character album to upload to train the individual image model of the first object (it also supports users to select photos to upload); customize the training of multiple theme template models to meet the image creation needs of users in different theme scenes; through the individual image model of the first object and the pre-trained theme template model, one-click creation is achieved, thereby obtaining a stable and high-quality photo image. The image generation scheme provided by the present disclosure will be described below with examples.

[0106] For example, based on the intelligently aggregated photos of people in the user's album, the user can directly select a person album and automatically select 20 to 30 high-quality photos through an algorithm. Alternatively, the user can manually select 20 to 30 photos of the same person to be used as image samples to generate an individual image model of the first object.

[0107] When selecting an existing portrait album, the selected images will be processed through face detection, face ratio detection, quality detection, occlusion detection, angle detection, human body detection, etc., in order to select high-quality photos for subsequent model training; at the same time, after the selection is completed, the user can still delete photos and continue to manually select other photos that meet the requirements. The manually selected photos will also be tested by the above algorithm to meet the standards that can be used for the subsequent generation of individual image models.

[0108] The selected photos of people that meet the standards are subjected to relevant detection, and the partial areas of the character features that need to be learned by the algorithm are cut out to improve the learning of the character features. The selected photos of people that meet the standards are subjected to data preprocessing, and the images after data preprocessing are used for model training to generate an individual image model of the first object. When generating the individual image model of the first object, two groups of image creation images based on the individual image model can also be automatically created to recommend to users, that is, two groups of image creation images of different theme scenes can be generated to recommend to users.

[0109] Optionally, the user can be provided with a variety of theme template models of different theme scenes, and the user can create images by selecting different theme scenes. Optionally, the user can choose to regenerate the generated results, and can also immersively preview a single image in the image creation image of the generated theme scene, and can also generate a high-definition large image for the single image, and can also save the generated image. Optionally, the image generation solution provided by the present disclosure can also provide users with a text image function, which can meet the text image creation needs of free input prompts.

[0110] Figure 4 is a block diagram of an image generating device according to an exemplary embodiment. Figure 4 , the device includes: a first providing module 401 and a display module 402.

[0111] The first providing module 401 is used to provide an image selection interface; the image selection interface is used to obtain the selected image to be processed.

[0112] The display module 402 is used to display the target image of the first object and / or at least one group of image creation images of the theme scene; the target image of the first object is an image generated based on the individual image model of the first object, and the image creation image of each group of theme scenes is an image generated based on the corresponding theme template model combined with the individual image model, and the individual image model is a model trained based on the image to be processed.

[0113] In some embodiments, Figure 5As shown, the device may also include a face detection module 503, an image processing module 504 and a first generation module 505. The face detection module 503 is used to perform face detection on the acquired image to be processed, and filter out the image of the first object from the image to be processed; the image processing module 504 is used to perform image processing on the image of the first object to obtain an image sample of the individual image model; the first generation module 505 is used to generate the individual image model of the first object based on the image sample. Figure 5 Medium 501-502 and Figure 4 401-402 have the same function and structure.

[0114] In one possible implementation, the first generation module 505 is specifically used to: perform inference on image samples to obtain description information of the image samples; add reference noise to the image samples to obtain a first mixed noise image; input the description information and the first mixed noise image into a preset extended model to obtain the predicted noise output by the extended model; adjust the network parameters of the extended model according to the loss function of the predicted noise and the reference noise until the model training end conditions are met, and determine the trained extended model as the individual image model of the first object.

[0115] In one possible implementation, image processing includes at least one of the following: face ratio detection processing, used to filter images whose face ratio is less than or equal to a first threshold; quality detection processing, used to filter images whose blur value is less than or equal to a second threshold; occlusion detection processing, used to filter images with occluded faces; angle detection processing, used to filter images whose face posture angles meet a first condition; and human body detection processing, used to filter images containing multiple human bodies.

[0116] In some embodiments, Figure 6 As shown, the device may also include: a first acquisition module 606, an inference module 607, an addition module 608, a second acquisition module 609, and a second generation module 610. The first acquisition module 606 is used to acquire a reference image from the image of the first object; the inference module 607 is used to infer the reference image to obtain the description information of the reference image; the addition module 608 is used to add reference noise to the reference image to obtain a second mixed noise image; the second acquisition module 609 is used to input the description information of the reference image and the second mixed noise image into the individual image model of the first object to obtain the predicted noise output by the individual image model; the second generation module 610 is used to generate the target image image of the first object according to the predicted noise and the second mixed noise image. Figure 6 501-605 and Figure 5 501-505 have the same function and structure.

[0117] In some embodiments, Figure 7 As shown, the device may further include: a fusion module 711 and a third generation module 712. The fusion module 711 is used to fuse at least one preset theme template model with the individual image model to obtain at least one fusion model; the third generation module 712 is used to generate at least one group of theme scene image creation images according to the image of the first object and at least one fusion model. Figure 7 701-710 and Figure 6 601-610 have the same function and structure.

[0118] In some embodiments, Figure 8 As shown, the device may further include: a second providing module 813 and a fourth generating module 814. The second providing module 813 is used to provide a theme template model selection interface; the theme template model selection interface is used to obtain the selected theme template model; the fourth generating module 814 is used to generate an image creation image corresponding to the theme scene according to the selected theme template model. Figure 8 801-812 and Figure 7 701-712 have the same function and structure.

[0119] In some embodiments, Fig. 9 As shown, the device may further include: a third providing module 915 and a fifth generating module 916. The third providing module 915 is used to provide an input interface; the input interface is used to obtain input text description information; the fifth generating module 916 is used to generate a corresponding image based on the text description information and the individual image model of the first object. Fig. 9 901-914 and Figure 8 801-814 have the same function and structure.

[0120] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0121] Fig.10 1 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 1000 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0122] Reference Figure 8, the electronic device 1000 may include one or more of the following components: a processing component 1002 , a memory 1004 , a power component 1006 , a multimedia component 1008 , an audio component 1010 , an input / output (I / O) interface 1012 , a sensor component 1014 , and a communication component 1016 .

[0123] The processing component 1002 generally controls the overall operation of the electronic device 1000, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1002 may include one or more processors 1020 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 1002 may include one or more modules to facilitate the interaction between the processing component 1002 and other components. For example, the processing component 1002 may include a multimedia module to facilitate the interaction between the multimedia component 1008 and the processing component 1002.

[0124] The memory 1004 is configured to store various types of data to support operations on the device 1000. Examples of such data include instructions for any application or method operating on the electronic device 1000, contact data, phone book data, messages, pictures, videos, etc. The memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0125] The power component 1006 provides power to the various components of the electronic device 1000. The power component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1000.

[0126] The multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1008 includes a front camera and / or a rear camera. When the device 1000 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0127] The audio component 1010 is configured to output and / or input audio signals. For example, the audio component 1010 includes a microphone (MIC), and when the electronic device 1000 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1004 or sent via the communication component 1016. In some embodiments, the audio component 1010 also includes a speaker for outputting audio signals.

[0128] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0129] The sensor assembly 1014 includes one or more sensors for providing various aspects of status assessment for the electronic device 1000. For example, the sensor assembly 1014 can detect the open / closed state of the device 1000, the relative positioning of components, such as the display and keypad of the electronic device 1000, and the sensor assembly 1014 can also detect the position change of the electronic device 1000 or a component of the electronic device 1000, the presence or absence of user contact with the electronic device 1000, the orientation or acceleration / deceleration of the electronic device 1000, and the temperature change of the electronic device 1000. The sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1014 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1014 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0130] The communication component 1016 is configured to facilitate wired or wireless communication between the electronic device 1000 and other devices. The electronic device 1000 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1016 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1016 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0131] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0132] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, and the instructions can be executed by the processor 1020 of the electronic device 1000 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0133] In an exemplary embodiment, a computer program product is also provided, comprising a computer program, which implements the above method when executed by the processor 1020 of 1000 .

[0134] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0135] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image generation method, characterized in that: include: Provide an image selection interface; The image selection interface is used to obtain the selected image to be processed; Displaying a target image of a first object and / or an image creation image of at least one group of subject scenes; The target image of the first object is an image generated based on the individual image model of the first object, and each group of image creation images of the theme scenes is an image generated based on the corresponding theme template model combined with the individual image model. The individual image model is a model trained based on the image to be processed.

2. The method according to claim 1, characterized in that The method further comprises: Performing face detection on the acquired image to be processed, and filtering out the image of the first object from the image to be processed; Performing image processing on the image of the first object to obtain an image sample of the individual image model; Based on the image samples, an individual image model of the first object is generated.

3. The method according to claim 2, characterized in that The step of generating the individual image model of the first object based on the image sample comprises: Performing reasoning on the image sample to obtain description information of the image sample; Adding reference noise to the image sample to obtain a first mixed noise image; Inputting the description information and the first mixed noise image into a preset extended model to obtain predicted noise output by the extended model; According to the loss function of the predicted noise and the reference noise, the network parameters of the extended model are adjusted until the model training end condition is met, and the trained extended model is determined as the individual image model of the first object.

4. The method according to claim 2 or 3, characterized in that The image processing includes at least one of the following: Face ratio detection processing, used to filter images whose face ratio is less than or equal to a first threshold; Quality detection processing, for filtering images whose fuzzy values ​​are less than or equal to a second threshold; Occlusion detection processing, used to filter images with occluded faces; Angle detection processing, for filtering images whose face posture angles meet the first condition; Human body detection processing is used to filter images containing multiple human bodies.

5. The method according to claim 1, characterized in that The method further comprises: acquiring a reference image from the image of the first object; Performing reasoning on the reference image to obtain description information of the reference image; Adding reference noise to the reference image to obtain a second mixed noise image; Inputting the description information of the reference image and the second mixed noise image into the individual image model of the first object to obtain the predicted noise output by the individual image model; A target image of the first object is generated according to the predicted noise and the second mixed noise image.

6. The method according to claim 1, characterized in that The method further comprises: Fusing at least one preset theme template model with the individual image model to obtain at least one fusion model; Based on the image of the first object and the at least one fusion model, an image creation image of the at least one group of subject scenes is generated.

7. The method according to claim 1, characterized in that The method further comprises: Providing a theme template model selection interface; the theme template model selection interface is used to obtain a selected theme template model; According to the selected theme template model, a creative image corresponding to the theme scene is generated.

8. The method according to claim 1, characterized in that The method further comprises: Providing an input interface; the input interface is used to obtain input text description information; Based on the text description information and the individual image model of the first object, a corresponding image is generated.

9. An image generating device, characterized in that: include: A first providing module is used to provide an image selection interface; The image selection interface is used to obtain the selected image to be processed; A display module, for displaying a target image of a first object and / or an image creation image of at least one group of subject scenes; The target image of the first object is an image generated based on the individual image model of the first object, and each group of image creation images of the theme scenes is an image generated based on the corresponding theme template model combined with the individual image model. The individual image model is a model trained based on the image to be processed.

10. An electronic device, characterized in that: include: processor; A memory for storing processor-executable instructions; wherein the instructions are executed by the processor to enable the processor to perform the method according to any one of claims 1 to 8.

11. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 8.

Images