Image generation method, device, equipment, storage medium and program product
By combining the facial extraction model and the literary and graphic model, images or videos are generated using facial features and text descriptions in the reference image, solving the problem of difficulty in style description in the prior art, and achieving the accuracy and consistency of image generation.
Patent Information
- Application Number
- CN202311860217.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-31
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-12-31
AI Technical Summary
Existing generative artificial intelligence technologies are difficult to accurately generate images of people or animals with specific looks that users want through text descriptions, because it is difficult for users to describe the desired style in text.
The facial features are obtained from the reference image through the facial extraction model, and combined with text description, the Wensheng Diagram model or Wensheng Video model generates images or videos with specific facial features to achieve the maintenance of the face image and character identity.
Users can extract facial features by entering a reference picture to generate images or videos that match the description, solving the problem of difficulty in style description and achieving the accuracy and consistency of image generation.
Smart Images

Figure CN118035492B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image generation technology, and in particular to an image generation method, device, equipment, storage medium and program product. Background Art
[0002] Generative AI (Artificial Intelligence Generated Content, AIGC) refers to the technology based on artificial intelligence technical methods such as generative adversarial networks and large pre-trained models, which generates relevant content with appropriate generalization capabilities through learning and recognition of existing data. The core idea of AIGC technology is to use artificial intelligence algorithms to generate content with certain creativity and quality. Through training models and learning from a large amount of data, AIGC can generate relevant content based on the text description entered by the user. However, the current AIGC technology can only generate images based on the text description entered by the user. When the user wants to generate an image of a person or animal with a certain appearance, it is difficult for the user to describe the style in words, so the style of the image generated by AIGC is often not what the user wants. Summary of the invention
[0003] One or more embodiments of the present specification provide an image generation method, apparatus, device, storage medium, and program product that can generate an AI image or AI video based on facial features extracted from an image.
[0004] In a first aspect, an embodiment of the present specification provides an image generation method, comprising: obtaining facial features according to a reference image specified by a user through a facial extraction model; obtaining a text description; generating an artificial intelligence image according to the facial features and the text description through a Vincent graph model, or generating an artificial intelligence video according to the facial features and the text description through a Vincent video model; the artificial intelligence image or the artificial intelligence video contains a character having the facial features.
[0005] Optionally, the facial features include at least one of the following: face shape information, skin information, facial features information, hair information, expression information, and head decoration information.
[0006] According to the image generation method provided in some embodiments of the present specification, the facial features are acquired according to a reference image specified by a user through a facial extraction model, comprising: acquiring a mapping space based on a reference image specified by a user through a mapping space model, the mapping space containing the facial features of the reference image, wherein the mapping space model includes vectors of multiple dimensions.
[0007] According to the image generation method provided by some embodiments of the present specification, the user-specified reference image contains at least two facial areas, and the method further includes: obtaining weights corresponding to the at least two facial areas respectively; obtaining facial features according to the user-specified reference image through a facial extraction model includes: obtaining facial features according to the user-specified reference image and the weights of the at least two facial areas through a facial extraction model, and the facial features are a fusion of the at least two facial areas according to the weights.
[0008] According to the image generation method provided by some embodiments of the present specification, the user-specified reference image contains at least two facial areas, and the acquiring facial features according to the user-specified reference image through a facial extraction model includes: acquiring facial features of the at least two facial areas according to the user-specified reference image through a facial extraction model; the method also includes: acquiring weights corresponding to the at least two facial areas respectively; generating an artificial intelligence image according to the facial features and the text description through a Vincent graph model, or generating an artificial intelligence video according to the facial features and the text description through a Vincent video model, includes: generating an artificial intelligence image according to the facial features corresponding to the at least two facial areas respectively, the weights, and the text description through a Vincent graph model; or generating an artificial intelligence video according to the facial features corresponding to the at least two facial areas respectively, the weights, and the text description through a Vincent video model.
[0009] According to the image generation method provided by some embodiments of the present specification, the user-specified reference image contains at least two facial areas, and the acquiring facial features according to the user-specified reference image through a facial extraction model includes: acquiring at least two facial features corresponding to the at least two facial areas respectively according to the user-specified reference image through a facial extraction model; generating an artificial intelligence image according to the facial features and the text description through a Vincent graph model, or generating an artificial intelligence video according to the facial features and the text description through a Vincent video model includes: generating an artificial intelligence image or an artificial intelligence video according to the at least two facial features and the text description through a Vincent graph / video model, and the artificial intelligence image or the artificial intelligence video includes at least two characters corresponding to the at least two facial features respectively.
[0010] According to the image generation method provided by some embodiments of the present specification, the user-specified reference image includes a reference image uploaded by the user and / or a template image selected by the user.
[0011] According to the image generation method provided by some embodiments of the present specification, the obtaining of the text description includes: generating multiple template descriptions; and obtaining the text description based on a template description randomly selected from the multiple template descriptions or a template description selected by a user.
[0012] According to the image generation method provided by some embodiments of the present specification, the obtaining of the text description includes: obtaining an initial text description input by a user; obtaining supplementary information based on the initial text description input by the user, wherein the supplementary information includes at least one of the following: scene information, composition information, character's action and expression information, clothing and accessories information; and determining the text description based on the initial text description and the supplementary information.
[0013] According to the image generation method provided by some embodiments of the present specification, the method also includes: obtaining style information according to a reference image specified by a user through a style extraction model; generating an artificial intelligence image according to the facial features and the text description through a Vincent graph model, or generating an artificial intelligence video according to the facial features and the text description through a Vincent video model, including: generating an artificial intelligence image according to the facial features, the text description and the style information through a Vincent graph model, or generating an artificial intelligence video according to the facial features, the text description and the style information through a Vincent video model.
[0014] In a second aspect, one or more embodiments of the present specification provide an image generating device, comprising: a facial extraction model, used to acquire facial features based on a reference image specified by a user; a first acquisition module, used to acquire a text description of the user; a text image / video model, used to generate an artificial intelligence image or an artificial intelligence video based on the facial features and the text description; the artificial intelligence image or the artificial intelligence video contains a character having the facial features.
[0015] In a third aspect, one or more embodiments of the present specification provide an image generating device, comprising: a processor and a memory, wherein the memory stores executable code, and when the executable code is processed by the processor, the processor can execute any one of the methods described.
[0016] In a fourth aspect, one or more embodiments of the present specification provide a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, enables the electronic device to execute any one of the methods described above.
[0017] In a fifth aspect, one or more embodiments of the present specification provide a computer program product, including a computer program or computer executable instructions, which implement any of the methods described when executed by a processor.
[0018] The embodiments of this specification use a model to extract the facial features of the target character. The user can extract the facial features by simply inputting a reference picture, without the need for training with a large number of pictures to extract the face of the character, so that the facial features and text description can be simultaneously input into the text-based image model, and an image of the target character with the facial features in the scene described by the user can be generated, thereby achieving facial image preservation or character identity preservation. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] This specification will be further described in the form of exemplary embodiments, which will be described in detail by way of the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same numbers represent the same structures or steps.
[0020] Figure 1 It is a flowchart of an embodiment of the image generation method of this specification.
[0021] Figure 2 It is a flowchart of another embodiment of the image generating method of the present specification.
[0022] Figure 3 It is a schematic block diagram of an embodiment of an image generating device of the present specification.
[0023] Figure 4 It is a schematic block diagram of another embodiment of the image generating device of the present specification.
[0024] Figure 5 It is a structural diagram of an embodiment of an image generating device in this specification. DETAILED DESCRIPTION
[0025] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the embodiments will be described in detail below with reference to the accompanying drawings. Obviously, the contents described below are some examples or embodiments of this specification. For ordinary technicians in this field, without paying creative work, the technical solutions or means disclosed in this specification can also be applied to other scenarios based on these technical contents.
[0026] It should be understood that the "system", "device", "unit" and / or "module" used in this specification is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.
[0027] Unless otherwise specified, technical terms used in this specification to describe components, elements, etc. do not refer to the singular, but may also include the plural. Generally speaking, terms such as "include", "comprise", etc. only indicate that the steps, elements, or components that have been clearly identified are included, and these steps, elements, and components do not constitute an exclusive list, such as the method or device described may also include other steps or components.
[0028] It should also be understood that the term "and / or" used in this specification refers to any or all possible combinations of one or more associated listed items. Although the terms "first", "second", "third", etc. may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this specification, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.
[0029] Flowcharts are used in this specification to illustrate the operation steps performed by the device or system of the relevant embodiments, but unless otherwise specified, the order used to describe these steps should not be understood as a limitation on the order in which the steps are performed. A person of ordinary skill in the art can adjust the order in which these steps are performed based on the knowledge and information conveyed by the embodiments of this specification, and the adjustment includes but is not limited to swapping the order, merging multiple steps, and splitting a certain step.
[0030] Figure 1 FIG. 1 is a flow chart of an embodiment of the image generation method of the present specification. Figure 1 As shown, the method may include the following steps.
[0031] Step S101, acquiring facial features according to a reference image specified by a user through a facial extraction model.
[0032] In some embodiments, there are multiple ways to obtain the user-specified reference image, for example, determining an image uploaded by the user as the user-specified reference image, or determining an image currently shot by the user as the user-specified reference image, or determining an image selected from multiple images stored by the user as the user-specified reference image.
[0033] Optionally, the number of reference images specified by the user may be one or at least two. The reference image specified by the user may contain only one facial region or at least two facial regions. In the example where the reference image specified by the user contains at least two facial regions, the facial extraction model randomly extracts facial features of a character's face, or may extract facial features of a face selected by the user. Alternatively, the facial extraction model may perform feature extraction on the face of the person in the reference image by default, or may perform feature extraction on the face of the animal by default. The facial extraction model may extract facial features of only one facial region and output them, or may extract facial features of multiple facial regions and output them, or may extract facial features of multiple facial regions and fuse them, and output the fused facial features.
[0034] The facial region in the reference image may be the face of a cartoon character, a virtual character, or a real character. For example, the reference image may be an image of a user's pet (eg, a dog or a cat).
[0035] Optionally, in an example where the user-specified reference image includes at least two facial regions, weights corresponding to the at least two facial regions are also obtained. Optionally, the facial extraction model further fuses facial features of the at least two facial regions according to the weights corresponding to the at least two facial regions, and then outputs the fused facial features.
[0036] Step S102, obtaining text description.
[0037] In some embodiments, the text description can be obtained by direct user input. Optionally, multiple template descriptions can also be provided to the user, and the template description selected by the user can be used as the text description. Providing multiple template descriptions can be generated online in real time under the trigger of the user, or it can be pre-trained and screened out relatively high-quality scene descriptions, and displayed to the user for selection under the trigger of the user. There are many ways to obtain template descriptions. For example, first generate 10,000 text descriptions through a language model, and then manually screen out 2,000 text descriptions with better output quality from the 10,000 text descriptions and store them. When the user clicks "AI One-click Generation" without input, the 2,000 text descriptions can be provided for the user to choose, or one or more of the 2,000 text descriptions can be randomly selected from the 2,000 text descriptions without user selection.
[0038] Optionally, the text description entered by the user can be obtained through the language model and then expanded and beautified, and the expanded and beautified text description can be input into the Wensheng graph model or Wensheng video model to generate better quality AI images or AI videos. There are many ways to expand and beautify. Optionally, the language model expands the text entered by the user while complying with the user's semantics, including expanding the subject itself in the text entered by the user, and / or expanding the description of environmental details. In some examples, the language model generates at least one of the following supplementary information to complete the expansion: scene information, composition information, character action expression information, clothing accessories information.
[0039] The scene information may include some common scene contents, such as whether it is indoors or outdoors, and some corresponding common scene elements. For example, if the user inputs that a girl is in a classroom, the language model may expand some common objects in the classroom, such as a blackboard, a desk, etc. The composition information may include whether the generated image is a distant view, a mid-range view, or a close-up view, the position of the character in the image, etc. Optionally, the composition information may be supplemented according to the composition rules and scene information corresponding to different scenes.
[0040] Step S103, generating an artificial intelligence image according to the facial features and the text description through a Vincent graph model, or generating an artificial intelligence video according to the facial features and the text description through a Vincent video model; the artificial intelligence image or the artificial intelligence video contains a character with the facial features.
[0041] For example, after extracting facial features from a reference image, the user generates multiple AI images of different styles, such as realistic style, cartoon style, etc. Optionally, the face of the character in the generated AI image may not be completely consistent with the character used to extract facial features in the reference image, as long as the main features are maintained, such as being similar in spirit; or they may be completely consistent, that is, the generated AI image is an image of the character used to extract facial features in the reference image in different scenes, different actions, and / or different expressions.
[0042] In an example in which a reference image specified by a user contains at least two facial regions, optionally, the facial extraction model outputs the fused facial features to a Vincent graph model or a Vincent video model, and the Vincent graph model or the Vincent video model generates an AI image or an AI video based on the fused facial features and a text description. Alternatively, optionally, the facial extraction model outputs at least two facial features to a Vincent graph model or a Vincent video model, and the Vincent graph model or the Vincent video model fuses the at least two facial features, and generates an AI image or an AI video based on the fused facial features and a text description. Alternatively, optionally, the facial extraction model outputs at least two facial features to a Vincent graph model or a Vincent video model, and the Vincent graph model or the Vincent video model further obtains weights corresponding to the at least two facial features, fuses the at least two facial features according to the weights, and generates an AI image or an AI video based on the fused facial features and a text description. Optionally, the weights corresponding to the at least two facial features can be input by the user. Alternatively, optionally, the facial extraction model outputs at least two facial features to a Vincent image model or a Vincent video model, which generates an AI image or AI video based on the at least two facial features and a text description, wherein the AI image or AI video contains at least two characters corresponding to the at least two facial features, respectively.
[0043] The current text-based graph model uses text descriptions to generate images. However, it is often difficult for users to describe what they want in words, for example, it is difficult for users to accurately describe a face. In some embodiments, the facial features of the target character are extracted by the model, and the user can extract the facial features by simply inputting a reference picture, without the need for training with a large number of pictures, so that the extracted facial features and text description can be input into the text-based graph model at the same time, and an image of the target character with the facial features in the scene described by the user can be generated, thereby achieving facial image preservation or character identity preservation.
[0044] Optionally, the face extraction model in each embodiment of the present specification can be obtained by training a mapping (embedding) space model for face extraction. The mapping space model can adopt the existing model architecture BIT. The training data of the mapping space model can include a variety of faces, such as faces of different people and / or different types of animals, etc. The mapping space model has good style learning ability, generalization ability and robustness, and can quickly learn to extract facial features from one or a few reference images. Optionally, the training data can also include images of the same face image under different lighting environments and / or different angles, which can improve the generalization ability of the model. When the reference image is input into the face extraction model, the face extraction model outputs a mapping space of vectors with multiple dimensions (for example, thousands of dimensions) based on the reference image to reflect the extracted facial features. The mapping space is input into the text image model or the text video model together with the text description.
[0045] There are many ways to train a face extraction model. In some examples, a face extraction model and a Vincent graph model or a Vincent video model can be trained together. Taking the Vincent graph model as an example, first construct an initial mapping space model and an initial Vincent graph model, and use multiple sample images with different faces as training samples. By inputting the training sample into the initial mapping space model, a sample mapping space is obtained, and then the sample mapping space and the sample text description are input into the Vincent graph model to generate an AI picture; finally, a loss function is constructed based on the similarity at the pixel level between the generated AI picture and the sample style image, and the similarity obtained by constructing the loss function by improving the similarity is used to train the optimized mapping space model, or the optimized mapping space model and the Vincent graph model are trained simultaneously. The initial Vincent graph model used in the training can be a ready-made Vincent graph model, or it can be a Vincent graph model that has been trained separately, which is not limited here.
[0046] In some examples, in addition to using a facial extraction model to obtain facial features from a reference image and inputting them into a Vincent image model or a Vincent video model, users can also use a style extraction model to obtain style information from a reference image and input it into a Vincent image model or a Vincent video model to generate AI images or videos with the same or similar style. Figure 2 FIG. 1 is a flow chart of another embodiment of the image generation method of the present specification. Figure 2 As shown, the method may include the following steps.
[0047] Step S201, acquiring facial features according to a facial reference image specified by a user through a facial extraction model.
[0048] For the explanation of step S201, please refer to the explanation of step S101, which will not be repeated here.
[0049] Step S202: acquiring style information according to a style reference image specified by a user through a style extraction model.
[0050] The style reference image input into the style extraction model and the face reference image input into the face extraction model may be the same or different. There may be multiple types of style information extracted. For example, the style information may include at least one of the following: color information, local texture information, material information, brushstroke information, lighting information, and composition information.
[0051] Optionally, the number of style reference images may be one or at least two. In the example where there are at least two style reference images, the style extraction model extracts and outputs the style information of each reference image, or the style extraction model may extract and fuse the style information of each reference image, and then output the fused style information. Optionally, the style weights of the at least two style reference images are obtained, and the style information of the at least two style reference images is fused according to the style weights. Optionally, the style weights of the two style reference images may be input by a user.
[0052] Step S203, obtaining text description.
[0053] For the explanation of step S203, please refer to the explanation of step S102, which will not be repeated here.
[0054] Step S204, generating an artificial intelligence image according to the facial features, the style information and the text description through a Vincent graph model, or generating an artificial intelligence video according to the facial features, the style information and the text description through a Vincent video model; the artificial intelligence image or the artificial intelligence video contains a character with the facial features.
[0055] When generating AI images or AI videos, users may want to obtain pictures of a specific style, such as cartoon style, anime style, photography style, movie style, etc. For example, the user may want a Hong Kong movie style, or even the style of Stephen Chow's Hong Kong movies, rather than the European and American Hollywood movie style; for example, the user may want a very abstract style of painting; but these styles are difficult to describe specifically and to express accurately through words. In some embodiments, the style information of the style reference image specified by the user is extracted through a style extraction model and input into the Vincent graph model or Vincent video, so that the user does not need to describe the style in words, but can obtain an AI image or AI video of a similar style by finding a reference image of a similar style.
[0056] Figure 3 FIG. 1 is a schematic block diagram of an embodiment of an image generating device of the present specification. Figure 3 As shown, the image generating device 300 includes the following modules.
[0057] The face extraction model 301 is used to obtain facial features according to a reference image specified by a user.
[0058] The first acquisition module 302 is used to acquire a text description.
[0059] The text image / video model 303 is used to generate an artificial intelligence image or an artificial intelligence video based on the facial features and the text description, wherein the artificial intelligence image or the artificial intelligence video contains a character with the facial features.
[0060] Optionally, the facial features include at least one of the following: face shape information, skin information, facial features information, hair information, expression information, and head decoration information.
[0061] In some embodiments, when the facial extraction model 301 acquires facial features based on a reference image specified by a user, it is specifically used to: acquire a mapping space based on a reference image specified by a user through a mapping space model, wherein the mapping space contains facial features of the reference image, wherein the mapping space model includes vectors of multiple dimensions.
[0062] In some embodiments, the reference image specified by the user may include at least two facial regions, and the apparatus may further include: a second acquisition module, configured to acquire weights corresponding to the at least two facial regions respectively.
[0063] When acquiring facial features according to a reference image specified by a user, the facial extraction model 301 is specifically used to: acquire facial features according to the reference image specified by the user and the weights of the at least two facial regions through the facial extraction model, wherein the facial features are a fusion of the at least two facial regions according to the weights.
[0064] In some embodiments, the reference image specified by the user may include at least two facial regions. When the facial extraction model 301 acquires facial features based on the reference image specified by the user, it is specifically used to: acquire facial features of the at least two facial regions based on the reference image specified by the user through the facial extraction model; the device also includes: a third acquisition module, used to acquire weights corresponding to the at least two facial regions respectively; when the Vincent graph / video model 303 generates an artificial intelligence image or an artificial intelligence video based on the facial features and the text description, it is specifically used to: generate an artificial intelligence image or an artificial intelligence video based on the facial features corresponding to the at least two facial regions and the weights, and the text description through the Vincent graph model.
[0065] In some embodiments, the user-specified reference image contains at least two facial regions, and acquiring facial features according to the user-specified reference image through a facial extraction model includes: acquiring facial features of the at least two facial regions through a facial extraction model according to the user-specified reference image; generating an artificial intelligence image according to the facial features and the text description through a Vincent graph model, or generating an artificial intelligence video according to the facial features and the text description through a Vincent video model includes: generating an artificial intelligence image or an artificial intelligence video according to the facial features corresponding to the at least two facial regions and the text description through a Vincent graph / video model, and the artificial intelligence image or the artificial intelligence video includes two characters having facial features corresponding to the at least two facial regions.
[0066] Optionally, the reference image specified by the user may include a reference image uploaded by the user and / or a template image selected by the user.
[0067] In some embodiments, when acquiring the text description, the first acquisition module 302 is specifically used to: generate multiple template descriptions; and acquire the text description according to a template description randomly selected from the multiple template descriptions or a template description selected by a user.
[0068] In some embodiments, when acquiring text descriptions, the first acquisition module 302 is specifically used to: acquire an initial text description input by a user; acquire supplementary information based on the initial text description input by the user, wherein the supplementary information includes at least one of the following: scene information, composition information, character's action and expression information, clothing and accessories information; determine the text description based on the initial text description and the supplementary information.
[0069] Figure 4 FIG. 1 is a schematic block diagram of an embodiment of an image generating device of the present specification. Figure 4 As shown, the image generating device 400 includes the following modules.
[0070] The face extraction model 401 is used to obtain facial features according to a face reference image specified by a user.
[0071] The style extraction model 402 is used to obtain style information according to a style reference image specified by a user.
[0072] The first acquisition module 403 is used to acquire a text description.
[0073] The text image / video model 404 is used to generate an artificial intelligence image or an artificial intelligence video based on the facial features, the style information and the text description, wherein the artificial intelligence image or the artificial intelligence video contains a character with the facial features.
[0074] Some embodiments of the present specification may also be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) on which an executable code (or a computer program or a computer instruction code) is stored. When the executable code (or a computer program or a computer instruction code) is executed by a processor of an electronic device (or a server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present specification.
[0075] Some embodiments of the present specification may also be implemented as a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, part or all of the steps of the above method of the present specification are implemented.
[0076] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. Figure 5 As shown, the electronic device 500 includes a processor 501 and a memory 502. The memory 502 is used to store a computer program. The processor 501 is used to execute the above-mentioned image generation method when calling the computer program. Further, the electronic device may also include a bus, a microphone, a speaker, a display and a camera. Among them, the processor 501, the memory 502, the microphone, the speaker, the display and the camera communicate through the bus, and can also communicate through other means such as wireless transmission.
[0077] It is understood that the processor in the embodiments of this specification may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.
[0078] The method steps in the embodiments of this specification can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.
[0079] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of this specification is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.
[0080] It should be understood that the various numerical numbers involved in the embodiments of this specification are only used for the convenience of description and are not used to limit the scope of the embodiments of this specification.
[0081] The embodiments of the present specification have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. An image generation method, characterized in that: include: Acquiring facial features according to a reference image specified by a user through a facial extraction model, which includes: after the reference image is input into the facial extraction model, the facial extraction model outputs a mapping space of vectors having multiple dimensions based on the reference image to reflect the extracted facial features; Obtaining a text description; it includes: obtaining an initial text description input by a user; completing the expansion by generating supplementary information according to the semantics of the initial text description input by the user through a language model, wherein the supplementary information includes at least one of the following: scene information, composition information, character action and expression information, clothing and accessories information, and the expansion includes extending the main body of the initial text description and / or extending the environmental details; determining the text description according to the initial text description and the supplementary information; and also includes: providing a plurality of template descriptions to the user, and using the template description selected by the user as the text description; wherein the plurality of template descriptions are generated online in real time under the triggering of the user, or a first number of template descriptions are generated in advance through a language model. text descriptions, then filtering out a second number of text descriptions with good image output quality from the first number of text descriptions and storing them, and when the user clicks to generate without inputting, providing the second number of text descriptions for the user to select, or randomly selecting one or more of the second number of text descriptions without the user selecting; the mapping space is input into a Vincent graph model or a Vincent video model together with the text descriptions, and an artificial intelligence image is generated according to the facial features and the text description by the Vincent graph model, or an artificial intelligence video is generated according to the facial features and the text description by the Vincent video model; the artificial intelligence image or the artificial intelligence video contains a character with the facial features; Among them, the face extraction model and the Vincent graph model or the Vincent video model are trained together, including: constructing an initial mapping space model and an initial Vincent graph model or an initial Vincent video model, inputting multiple faces of different people and / or different types of animals, and / or sample images of the same face image under different lighting environments and / or different angles as training samples into the initial mapping space model to obtain a sample mapping space; inputting the sample mapping space and the sample text description into the initial Vincent graph model or the initial Vincent video model to generate an artificial intelligence image or an artificial intelligence video; constructing a loss function based on the similarity between the artificial intelligence image or the artificial intelligence video and the sample image at the pixel level, training and optimizing the initial mapping space model by improving the similarity to construct the loss function, or synchronously training and optimizing the initial mapping space model and the initial Vincent graph model or the initial Vincent video model to generate the face extraction model and the Vincent graph model or the Vincent video model.
2. The method according to claim 1, characterized in that The facial features include at least one of the following: Face shape information, skin information, facial features information, hair information, expression information, and head decoration information.
3. The method according to claim 1, characterized in that The method of acquiring facial features according to a reference image specified by a user through a facial extraction model includes: A mapping space is acquired based on a reference image specified by a user through a mapping space model, wherein the mapping space contains facial features of the reference image, wherein the mapping space model includes vectors of multiple dimensions.
4. The method according to claim 1, characterized in that: The user-specified reference image contains at least two facial regions, and the method further comprises: Obtaining weights corresponding to the at least two facial regions respectively; The method of acquiring facial features according to a reference image specified by a user through a facial extraction model includes: Facial features are acquired through a facial extraction model according to a reference image specified by a user and weights of the at least two facial regions, wherein the facial features are a fusion of the at least two facial regions according to the weights.
5. The method according to claim 1, characterized in that The user-specified reference image contains at least two facial regions, The acquiring facial features according to the reference image specified by the user through the facial extraction model comprises: acquiring facial features of the at least two facial regions according to the reference image specified by the user through the facial extraction model; The method further comprises: Obtaining weights corresponding to the at least two facial regions respectively; The step of generating an artificial intelligence image according to the facial features and the text description by using a Vincent image model, or generating an artificial intelligence video according to the facial features and the text description by using a Vincent video model, includes: Generate an artificial intelligence image using a text graph model according to the facial features corresponding to the at least two facial regions, the weights, and the text description; or, An artificial intelligence video is generated through a Wensheng video model according to the facial features corresponding to the at least two facial regions, the weights, and the text description.
6. The method according to claim 1, characterized in that The user-specified reference image contains at least two facial regions, The acquiring facial features according to the reference image specified by the user through the facial extraction model comprises: acquiring at least two facial features respectively corresponding to the at least two facial regions according to the reference image specified by the user through the facial extraction model; The step of generating an artificial intelligence image according to the facial features and the text description by using a Vincent image model, or generating an artificial intelligence video according to the facial features and the text description by using a Vincent video model, includes: Generate an artificial intelligence image according to the at least two facial features and the text description by using a text graph model, wherein the artificial intelligence image includes at least two characters corresponding to the at least two facial features respectively; or An artificial intelligence video is generated according to the at least two facial features and the text description through a Wensheng video model, and the artificial intelligence video includes at least two characters corresponding to the at least two facial features respectively.
7. The method according to claim 1, characterized in that The user-specified reference image includes a reference image uploaded by the user and / or a template image selected by the user.
8. The method according to claim 1, characterized in that The obtaining of text description includes: Generate multiple template descriptions; The text description is obtained according to a template description randomly selected from the multiple template descriptions or a template description selected by a user.
9. The method according to claim 1, characterized in that: The method further comprises: Acquire style information based on the reference image specified by the user through the style extraction model; The step of generating an artificial intelligence image according to the facial features and the text description by using a Vincent image model, or generating an artificial intelligence video according to the facial features and the text description by using a Vincent video model, includes: Generate an artificial intelligence image according to the facial features, the text description and the style information through a text-based image model, or, Generate an artificial intelligence video based on the facial features, the text description and the style information through a Vincent video model.
10. An image generating device, characterized in that: include: A face extraction model, used for acquiring facial features according to a reference image specified by a user, comprising: when the reference image is input into the face extraction model, the face extraction model outputs a mapping space of vectors having multiple dimensions based on the reference image to reflect the extracted facial features; The first acquisition module is used to acquire the text description of the user; the first acquisition module is further used to: acquire the initial text description input by the user; generate supplementary information to complete the expansion according to the semantics of the initial text description input by the user through a language model, wherein the supplementary information includes at least one of the following: scene information, composition information, character action and expression information, clothing and accessories information, and the expansion includes an extended description of the main body of the initial text description and / or an extended description of the environmental details; determine the text description according to the initial text description and the supplementary information; the first acquisition module is further used to provide a plurality of template descriptions to the user, and use the template description selected by the user as the text description, wherein the plurality of template descriptions The text descriptions are generated online in real time under the trigger of the user, or a first number of text descriptions are generated in advance through a language model, and then a second number of text descriptions with good image output quality are screened out from the first number of text descriptions and stored. When the user clicks to generate without input, the second number of text descriptions are provided for the user to select, or one or more text descriptions are randomly selected from the second number of text descriptions without user selection; the mapping space is input into a Vincent image / video model together with the text description, and the Vincent image / video model is used to generate an artificial intelligence image or an artificial intelligence video according to the facial features and the text description; the artificial intelligence image or the artificial intelligence video contains a character with the facial features; Among them, the face extraction model and the Vincent map / video model are trained together, including: constructing an initial mapping space model and an initial Vincent map / video model, inputting multiple faces of different people and / or different types of animals, and / or sample images of the same face image under different lighting environments and / or different angles as training samples into the initial mapping space model to obtain a sample mapping space; inputting the sample mapping space and sample text description into the initial Vincent map / video model to generate an artificial intelligence image or an artificial intelligence video; constructing a loss function based on the similarity between the artificial intelligence image or artificial intelligence video and the sample image at the pixel level, and training and optimizing the initial mapping space model by improving the similarity to construct the loss function, or synchronously training and optimizing the initial mapping space model and the initial Vincent map / video model to generate the face extraction model and the Vincent map / video model.
11. An image generating device, characterized in that: include: A processor and a memory, wherein the memory stores executable code, and when the executable code is processed by the processor, the processor can execute the method as claimed in any one of claims 1 to 9.
12. A computer-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the electronic device to execute the method according to any one of claims 1 to 9.
13. A computer program product comprising a computer program or computer executable instructions, characterized in that: When the computer program or computer executable instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.
Citation Information
Patent Citations
Virtual content generation method and device, electronic equipment and storage medium
CN114904270A
Method for generating head portrait and method for replacing head portrait
CN116110099A
Video generation method and server
CN116233491A
Video generation method and device, model training method and device, equipment and medium
CN116320216A
Image generation method and device for PowerPoint
CN116452706A
