Image generation method and related device

Abstract

The invention provides an image generation method and a related device. The electronic device can receive and respond to a first input, and determine a first scene template model and a first cue word in a first shooting scene. And obtaining a personal image training set, and training the first portrait model through the personal image training set. Then, a first front face image is obtained from the personal image training set, and a first face key point graph is generated based on the first front face image; then, obtaining a target template image, and generating a first skeleton key point graph based on the target template image; then, face key points in the first skeleton key point graph are replaced with face key points in the first face key point graph, and a posture control graph is obtained. And finally, generating a personalized image based on the attitude control chart, the first portrait model, the first scene template model and the first cue word through a text graph model. And the picture effect of the portrait and the background in the generated image with the specified style is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation method and related devices. Background Art

[0002] In daily life, users often use images containing portraits, including more formal portraits and ID photos, as well as casual artistic portraits, etc. In order to take ID photos with relevant backgrounds or clothing in a timely manner as required, products that can quickly generate images are generally used to generate images of corresponding scenes, but the generated images are not good. Summary of the invention

[0003] The present application provides an image generation method and related devices, which first control a text graph model through a posture control graph, a portrait model of a specified user, and a first scene template model to generate an image of a specified style based on prompt words, thereby improving the picture effects of the portrait and background in the generated image of the specified style.

[0004] In a first aspect, the present application provides an image generation method, including: an electronic device can receive a first input. In response to the first input, the electronic device can determine a first scene template model and a first prompt word in a first shooting scene. The electronic device can obtain a personal image training set, and the personal image training set includes multiple personal images of a specified user. The electronic device can train a first portrait model through the personal image training set, and the first portrait model is used to generate facial features of the specified user. The electronic device can obtain a first frontal face image from the personal image training set, and generate a first face key point map based on the first frontal face image, the first frontal face image includes a face of the specified user in a frontal face posture, and the first face key point map includes face key points in the first frontal face image. The electronic device can obtain a target template image, and generate a first skeletal key point map based on the target template image, the target template image includes a face of a model when posing a specified posture, and the first skeletal key point map includes face key points and limb key points in the target template image. The electronic device can replace the face key points in the first skeletal key point map with the face key points in the first face key point map to obtain a posture control map. The electronic device can use the gesture control graph, the first portrait model and the first scene template model to control the Wensheng graph model to generate a personalized image based on the first prompt word, and the personalized image includes the face of a specified user when posing a specified gesture in a first shooting scene.

[0005] In this way, the face in the generated personalized image has a high similarity with the face of the specified user, ensuring high fidelity. At the same time, the clothing and background generated in the personalized image that meet the user's needs can be controlled.

[0006] In a possible implementation, the method further includes: the electronic device can extract a facial region of a portrait in the personalized image to obtain a facial mask image, where the facial mask image is used to characterize a first facial region of the portrait in the personalized image. The electronic device can use the facial mask image, the posture control image, the first portrait model, and the first scene template model control image model to perform fine restoration on the face of the designated user in the personalized image to obtain a target image.

[0007] In this way, the similarity between the face and the face of the designated user in the obtained target image is further improved, ensuring the fidelity of the facial contour and fine features of the facial features in the target image.

[0008] In a possible implementation, the similarity between the face in the target image and the face in the first frontal face image is greater than the similarity between the face in the personalized image and the face in the first frontal face image.

[0009] In a possible implementation, the method further includes: the electronic device may set the model weight of the first portrait model in the process of the Vincent graph model generating the personalized image to a first weight. The electronic device may set the model weight of the first scene template model in the process of the Vincent graph model generating the personalized image to a second weight. In this way, by adjusting the first weight and the second weight, the background of the generated personalized image, the clothing of the characters in the personalized image, the face of the characters in the personalized image, etc. can be controlled to obtain a personalized image that better meets the user's needs.

[0010] In a possible implementation, the method further includes: the electronic device may set the model weight of the first portrait model in the process of generating the target image from the image-generated image model to a third weight. The electronic device may set the model weight of the first scene template model in the process of generating the target image from the image-generated image model to a fourth weight, and the fourth weight is less than the third weight. In this way, the generated target image can be controlled by adjusting the third weight and the fourth weight, so that the similarity between the face of the person in the target image and the face of the specified user is greatly improved, further improving the fidelity of the target image and improving the user experience.

[0011] In a possible implementation manner, the first weight is smaller than the third weight, and the second weight is larger than the fourth weight.

[0012] In a possible implementation manner, the first weight is equal to the second weight.

[0013] In a possible implementation, the method further includes: the electronic device can use the affine transformation matrix to perform an affine transformation on the first frontal face image to obtain a second frontal face image, in which the size and position of the face in the second frontal face image are the same as the size and position of the model's face in the target template image. The electronic device can detect the edge of the face in the second frontal face image to obtain a frontal face edge map. The electronic device can determine the position of the eyebrows in the frontal face edge map. The electronic device can intercept the edge of the portion above the eyebrows from the frontal face edge map to obtain a forehead edge control map. The electronic device can use the posture control map, the forehead edge control map, the first portrait model and the first scene template model to control the Vincent map model to generate a personalized image based on the first prompt word. In this way, the portrait in the personalized image can be made more vivid, and the face and head of the portrait in the personalized image can be made to fit more closely.

[0014] In a possible implementation, the facial mask image, the posture control image, the first portrait model, and the first scene template model control image raw image model are used to finely repair the face of the designated user in the personalized image to obtain a target image, specifically including: using the facial mask image, the forehead edge control image, the posture control image, the first portrait model, and the first scene template model control image raw image model to finely repair the face of the designated user in the personalized image to obtain the target image. In this way, the similarity between the face in the obtained target image and the face of the designated user is further improved.

[0015] In a possible implementation, extracting the facial region of the portrait in the personalized image to obtain a facial mask map specifically includes: segmenting the facial region of the portrait in the personalized image to obtain a preliminary mask map, where the preliminary mask map can be used to characterize a second facial region of the portrait in the personalized image, and expanding the preliminary mask map at a specified ratio to obtain the facial mask map.

[0016] In a possible implementation, the facial key points in the first skeleton key point map are replaced with the facial key points in the first facial key point map to obtain a posture control map, which specifically includes: obtaining a target template image, and generating a second facial key point map based on the target template image, the second facial key point map including the facial key points in the target template image. Determine an affine transformation matrix between the first facial key point map and the second facial key point map, the affine transformation matrix is ​​used to characterize the transformation relationship between the size and position of the facial key points in the first facial key point map and the facial key points in the second facial key point map. Use the affine transformation matrix to perform an affine transformation on the first facial key point map to obtain a third facial key point map. Remove the facial key points in the first skeleton key point map, retain the limb key points, and obtain a second skeleton key point map. Fuse the facial key points in the third facial key point map into the second skeleton key point map to obtain the posture control map.

[0017] In a possible implementation, the training of the first portrait model by the personal image training set specifically includes: adjusting the size of the image in the personal image training set to a specified size. The image in the personal image training set adjusted to the specified size is used to train the portrait model of the specified user. In this way, it is more convenient to train the first portrait model.

[0018] In a second aspect, the present application provides an electronic device comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store a computer program, so that when the one or more processors execute the computer program, the method in any possible implementation of the above-mentioned first aspect is executed.

[0019] In a third aspect, the present application provides another electronic device, comprising one or more functional modules, wherein the one or more functional modules are used to execute the method in any possible implementation of the first aspect above.

[0020] In a fourth aspect, the present application provides a chip system, comprising a processing circuit and an interface circuit, wherein the interface circuit is used to receive instructions and transmit them to the processing circuit, and the processing circuit is used to run instructions to execute a method in any possible implementation of any of the above aspects.

[0021] In a fifth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a processor of an electronic device, enables execution of a method in any possible implementation of the above-mentioned first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of the structure of a server provided in an embodiment of the present application;

[0024] Figure 3A-3G A set of interface schematic diagrams provided for embodiments of the present application;

[0025] Figure 4A-4F Another set of interface schematic diagrams provided for embodiments of the present application;

[0026] Figure 5 A schematic diagram of a flow chart of an image generation method provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of a method for determining an affine transformation matrix provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of an image processing method provided in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of a method for determining a posture control graph provided in an embodiment of the present application;

[0030] Fig. 9 A schematic diagram of another image processing method provided in an embodiment of the present application;

[0031] Fig.10 A schematic diagram of an image generation method provided in an embodiment of the present application;

[0032] Figure 11A-11B A schematic diagram of a model training method provided in an embodiment of the present application;

[0033] Fig.12 A schematic diagram of another image processing method provided in an embodiment of the present application;

[0034] Fig.13 A schematic diagram of another method for generating an image provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0036] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0037] The following introduces the hardware structure of an electronic device provided in an embodiment of the present application.

[0038] Figure 1 A schematic structural diagram of an electronic device 100 provided in an embodiment of the present application is shown.

[0039] It should be understood that Figure 1 The electronic device 100 shown is only an example, and the electronic device 100 may have more Figure 1 More or fewer components may be shown, two or more components may be combined, or there may be a different configuration of components. Figure 1 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0040] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a gyroscope sensor 180B, an acceleration sensor 180E, and a touch sensor 180K, etc.

[0041] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0042] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0043] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0044] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0045] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0046] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0047] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0048] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0049] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0050] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0051] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0052] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display screen panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), miniled, microled, micro-oled, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0053] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0054] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0055] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0056] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0057] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0058] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0059] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0060] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0061] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0062] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0063] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0064] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0065] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0066] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0067] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0068] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0069] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0070] The button 190 includes a power button, a volume button, etc. The button 190 may be a mechanical button. It may also be a touch button. The electronic device 100 may receive a button input and generate a key signal input related to the user settings and function control of the electronic device 100. The motor 191 may generate a vibration prompt. The SIM card interface 195 is used to connect a SIM card. The SIM card may be connected to or disconnected from the electronic device 100 by inserting the SIM card interface 195 or pulling the SIM card interface 195 out.

[0071] In the embodiment of the present application, the device type of the electronic device 100 can be a mobile phone, a tablet computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), as well as smart home devices such as smart large screens and smart speakers, wearable devices such as smart bracelets, smart watches, and smart glasses, extended reality (extended reality, XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), vehicle-mounted devices or smart city devices, and the like.

[0072] The following introduces the hardware structure of a server provided in an embodiment of the present application.

[0073] Figure 2 A schematic diagram of the hardware structure of a server provided in an embodiment of the present application is shown.

[0074] like Figure 2 As shown, the server 200 may include a communication interface 201 and a processor 202. Optionally, the server 200 may also include a memory 203. Among them, the memory 203 can be set inside the server 200, and can also be set outside the server 200. The functions implemented by the server 200 in the above embodiments can all be implemented by the processor 202. The processor 202 receives the data stream through the communication interface 201, and is used to implement the image generation method of any of the above embodiments. During the implementation process, each step of the processing flow can be completed by the hardware integrated logic circuit in the processor 202 or the instruction in the form of software to complete the image generation method of any of the above embodiments. For the sake of brevity, it will not be repeated here. The program code executed by the processor 202 for implementing the image generation method of any of the above embodiments for model training can be stored in the memory 203. The memory 203 is coupled to the processor 202.

[0075] The processor involved in the embodiments of the present application may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0076] The coupling in the embodiments of the present application is an indirect coupling or communication connection between devices, modules or modules, which can be electrical, mechanical or other forms, and is used for information exchange between devices, modules or modules.

[0077] The processor may operate in conjunction with a memory. The memory may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), or a volatile memory, such as a random-access memory (RAM). The memory is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0078] The specific connection medium between the above-mentioned communication interface, processor and memory is not limited in the embodiments of the present application. For example, the memory, processor and communication interface may be connected via a bus. The bus may be divided into an address bus, a data bus, a control bus, etc.

[0079] The embodiment of the present application provides an image generation method. The electronic device can receive and respond to a first input to determine a first scene template model and a first prompt word in a first shooting scene. A personal image training set is obtained, and a first portrait model is trained through the personal image training set. Then, a first frontal face image is obtained from the personal image training set, and a first face key point map is generated based on the first frontal face image. Then, a target template image is obtained, and a first skeleton key point map is generated based on the target template image. Then, the face key points in the first skeleton key point map are replaced with the face key points in the first face key point map to obtain a posture control map. Finally, a personalized image is generated based on the posture control map, the first portrait model, the first scene template model and the first prompt word through a Wensheng graph model. The method can control the background and clothing of the image generation, and at the same time, improve the similarity between the facial features and the face shape and the user himself, so that the generated image is more realistic, has higher fidelity, and improves the user experience effect.

[0080] The following describes an image generation method involved in an embodiment of the present application in detail in combination with application scenarios.

[0081] In some application scenarios, the electronic device 100 stores multiple portrait albums in a gallery, and the same portrait album contains multiple images of the same user. The electronic device 100 can select multiple portrait images of a specified user from the portrait album of the specified user based on the user's input. Then, the electronic device 100 can select a template image for generating a target image based on the user's input. The electronic device 100 can generate a target image of a specified style based on the multiple portrait images of the specified user and the template image. The style of the target image may include a ID photo style, an image photo style, an artistic photo style, and the like. Therefore, the electronic device 100 can generate target images of different scenes anytime and anywhere according to the needs of the user.

[0082] In a possible implementation, the electronic device 100 may select multiple portrait images of the specified user from the portrait album of the specified user based on the user's input. Then, the electronic device 100 may select a template image for generating a target image based on the user's input. The electronic device 100 may send the multiple portrait images of the specified user and the template image to the server 200. The server 200 may generate the target image based on the multiple portrait images of the specified user and the template image. The server 200 may return the target image to the electronic device 100.

[0083] For example, Figure 3AAs shown, the electronic device 100 can display a home screen interface 310. The interface 310 displays a page with application icons, which includes multiple application icons (e.g., weather application icon, stock application icon, calculator application icon, setting application icon, mail application icon, gallery application icon 311, etc.).

[0084] The electronic device 100 may receive an input (eg, a click) from a user on the gallery application icon 311. In response to the input, the electronic device 100 may display a Figure 3B The album interface 320 shown. The album interface 320 includes one or more albums, such as album 321.

[0085] The electronic device 100 may receive an input (eg, a click) from a user on the photo album 321. In response to the input, the electronic device 100 may display the following information: Figure 3C The image interface 330 shown in FIG. 30 may include multiple images and function controls 331 of a specified user.

[0086] The electronic device 100 may receive an input (eg, a click) from a user on the function control 331. In response to the input, the electronic device 100 may display a function control 331 on the image interface 330. Figure 3D The function selection page 336 shown. The function selection page 336 may include multiple personalized image function controls (e.g., ID photo control 332, image photo control 333, art photo control 334, more controls 335, etc.). Among them, the ID photo control 332 can be used to trigger the electronic device 100 to generate a target image in the style of an ID photo. The image photo control 333 can be used to trigger the electronic device 100 to generate a target image in the style of an image photo. The art photo control 334 can be used to trigger the electronic device 100 to generate a target image in the style of an art photo. The more controls 335 can be used to trigger the electronic device 100 to display other personalized image function controls.

[0087] The electronic device 100 may receive an input (eg, a click) from a user on the image control 333, and in response to the input, the electronic device 100 may display a Figure 3E The portrait template interface 340 shown in FIG. The portrait template interface 340 may include prompt information 345 , a generation control 343 , and multiple template images (eg, template image 341 and template image 342 , etc.).

[0088] The electronic device 100 may receive an input (e.g., a click) from a user on the template image 341. In response to the input, the template image 341 is in a selected state. The electronic device 100 may receive an input (e.g., a click) from a user on the generation control 343. In response to the input, the electronic device 100 may display a waiting prompt 346 on the image template interface 340, such as Figure 3F After waiting for a period of time, the electronic device 100 may display Figure 3G The portrait photo generation interface 350 shown may include a portrait photo 351 and a save control 352. The portrait photo 351 is a target image of a portrait style. The save control 352 may trigger the electronic device 100 to save the portrait photo 351.

[0089] In a possible implementation, the electronic device 100 may receive an input (e.g., a click) from a user on the generation control 343. In response to the input, the electronic device 100 may send multiple images of a specified user and the template image 341 to the server 200. The server 200 may generate a portrait photo 351 in response to the input of the electronic device 100, and send the portrait photo 351 to the electronic device 100. The electronic device 100 displays the image 351 as shown in FIG. Figure 3G The image generation interface 350 is displayed.

[0090] In some application scenarios, the electronic device 100 may be installed with an image editing application. The electronic device 100 may select a template image in the image editing application based on the user's input. The electronic device 100 may upload multiple portrait images of a specified user based on the user's input. The electronic device 100 may generate a target image of a specified style based on the multiple portrait images of the specified user and the template image. The style of the target image may include a ID photo style, an image photo style, an artistic photo style, and the like. Therefore, the electronic device 100 may generate target images of different scenes anytime and anywhere according to the needs of the user.

[0091] For example, Figure 4A As shown, the electronic device 100 may display a home screen interface 310. A page with application icons is displayed in the interface 310. The page includes multiple application icons, for example, an image editing application icon 411 and the like.

[0092] The electronic device 100 may receive an input (eg, a click) from a user on the image editing application icon 411, and in response to the input, the electronic device 100 may display a Figure 4BThe functional interface 420 shown. The functional interface 420 may include a plurality of personalized image function controls (e.g., a document photo control 421, an image photo control 422, an art photo control 423, more controls 424, etc.). Among them, the document photo control 421 may be used to trigger the electronic device 100 to generate a target image in the style of a document photo. The image photo control 422 may be used to trigger the electronic device 100 to generate a target image in the style of an image photo. The art photo control 423 may be used to trigger the electronic device 100 to generate a target image in the style of an art photo. The more controls 424 may be used to trigger the electronic device 100 to display other personalized image function controls.

[0093] The electronic device 100 may receive an input (eg, a click) from a user on the image control 422, and in response to the input, the electronic device 100 may display a Figure 4C The portrait template interface 430 shown in FIG. 4 may include prompt information 436 , an upload image control 431 , and multiple template images (eg, template image 432 and template image 433 , etc.).

[0094] The electronic device 100 may receive an input (e.g., a click) from a user on the template image 432. In response to the input, the template image 432 is in a selected state. The electronic device 100 may receive an input (e.g., a click) from a user on the upload image control 431. In response to the input, the electronic device 100 may display the following: Figure 4D The upload image interface 440 shown in FIG. 4 may include multiple images of a specified user and an upload control 441 .

[0095] The electronic device 100 may receive an input (e.g., a click) from a user on the multiple images of the specified user. In response to the input, the multiple images of the specified user are in a selected state. The electronic device 100 may receive an input (e.g., a click) from a user on the upload control 441. In response to the input, the electronic device 100 may display a waiting prompt 442 on the upload image interface 440, such as Figure 4E After waiting for a period of time, the electronic device 100 may display Figure 4F The portrait photo generation interface 450 shown in the figure may include a portrait photo 451 and a save control 452. The portrait photo 351 is a target image of a portrait style. The save control 452 may trigger the electronic device 100 to save the portrait photo 451.

[0096] An image generation method provided in an embodiment of the present application is introduced below.

[0097] Figure 5A flow chart of an image generating method provided in an embodiment of the present application is shown.

[0098] The method can be applied to the electronic device 100 or the server 200, and is not limited here. The embodiment of the present application takes the electronic device 100 as an example to introduce the image generation method.

[0099] like Figure 5 As shown, the method comprises the following steps:

[0100] S501. Obtain a personal image training set, where the personal image training set includes multiple personal images of a specified user.

[0101] The electronic device 100 may use the multiple personal images of the specified user in the album as the personal image training set based on the user's input. For example, the multiple personal images of the specified user may be the above Figure 3C The multiple images shown in , or the above Figure 4D Multiple images shown in .

[0102] S502: Train a portrait model of a designated user using a personal image training set.

[0103] The electronic device 100 may adjust the size of the images in the personal image training set to the same size. The electronic device 100 may use the images of the same size after adjustment to train the portrait model of the designated user. The portrait model of the designated user is used to learn the features of the face of the designated user from the image of the designated user. The trained portrait model of the designated user may include the features of the face of the designated user.

[0104] The portrait model of the designated user may be referred to as a first portrait model.

[0105] S503, select a personal frontal face image from the personal image training set, and detect the key points of the face in the personal frontal face image to obtain the key points of the face Figure 1 .

[0106] The electronic device 100 may select a personal frontal face image from the personal image training set, wherein the personal frontal face image includes a frontal face posture of a designated user, that is, the face posture of the designated user in the personal image is facing forward.

[0107] like Figure 6 As shown, the electronic device 100 can detect the key points of the face (such as key point 601, key point 602, etc.) in the frontal face image of the individual, and obtain the key points of the face. Figure 1. Among them, the key points of the face may include the key points of the facial features of the specified user (for example, the key points of the mouth, the key points of the ears, the key points of the nose, the key points of the eyes, and the key points of the eyebrows, etc.) and the key points of the face shape. The key points of the face shape are used to characterize the outline of the face of the specified user. The key points of the mouth are used to characterize the mouth features of the specified user. The key points of the ears are used to characterize the ear features of the specified user. The key points of the nose are used to characterize the nose features of the specified user. The key points of the eyes are used to characterize the eye features of the specified user. The key points of the eyebrows are used to characterize the eyebrow features of the specified user. For example, the key points of the face shape may include Figure 6 Key points of the face Figure 1 The key points shown are 601. The eyebrow key points may include Figure 6 Key points of the face Figure 1 Key points 602 are shown.

[0108] The above-mentioned personal frontal face image can be referred to as the first frontal face image.

[0109] The above facial key points Figure 1 It can be called the first face key point map.

[0110] S504: Obtain a target template image, and detect key points of the face in the target template image to obtain key points of the face. Figure 2 .

[0111] The electronic device 100 includes a plurality of template images. Figure 3F The template image 341 and the template image 342 shown in FIG. ...2 and the template image 341 and the template image 342 shown in FIG. Figure 4C The template image 432 and the template image 433 shown in FIG. The template image includes the frontal face posture of the model. The electronic device 100 can select a target template image from multiple template images. For example, the above Figure 3F As shown in , the electronic device 100 can select the template image 341 as the target template image in response to the user's input (eg, single click). Figure 4C As shown in , the electronic device 100 can select the template image 432 as the target template image in response to a user input (eg, a single click).

[0112] like Figure 6 As shown, the electronic device 100 can detect the key points of the face in the target template image and obtain the key points of the face. Figure 2. Among them, the key points of the face may include the key points of the model's facial features (for example, the mouth key points, the ear key points, the nose key points, the eye key points and the eyebrow key points, etc.) and the face shape key points. The face shape key points are used to characterize the outline of the model's face in the target template image. The mouth key points are used to characterize the model's mouth features. The ear key points are used to characterize the model's ear features. The nose key points are used to characterize the model's nose features. The eye key points are used to characterize the model's eye features. The eyebrow key points are used to characterize the model's eyebrow features. For example, the face shape key points may include Figure 6 Key points of the face Figure 2 Key points 603 are shown. Eyebrow key points may include Figure 6 Key points of the face Figure 2 Key points 604 are shown.

[0113] In an embodiment of the present application, the above-mentioned method of detecting facial key points includes but is not limited to using a deep convolutional network or a cascaded deep neural network, etc.

[0114] The above facial key points Figure 2 It can be called the second face key point map.

[0115] S505: Determine key points of the face Figure 1 And facial key points Figure 2 The affine transformation matrix between .

[0116] like Figure 6 As shown, the electronic device 100 can be configured based on the key points of the face. Figure 1 Face key points and face key points Figure 2 The correspondence between the key points of the face in the image is used to determine the key points of the face. Figure 1 And facial key points Figure 2 The affine transformation matrix between . For example, the key points of a face Figure 1 The middle face feature key point 601 corresponds to the face key point Figure 2 Middle face feature key points 603. Face key points Figure 1 The middle eyebrow key point 602 corresponds to the face key point Figure 2 The eyebrow key point 604. The affine transformation matrix is ​​used to represent the key points of the face Figure 1 Specify the key points and face key points of the user's face Figure 2 The transformation relationship between the size and position of the key points of the model's face.

[0117] S506: Detect the skeleton key points in the target template image to obtain the skeleton key points Figure 1 .

[0118] like Figure 7As shown, the electronic device 100 can detect the skeleton key points of the model in the target template image, and the skeleton key points include face key points and limb key points. Figure 1 The skeleton key points are included in the target template image. The skeleton key points are used to mark the skeleton positions of the model in the target template image, such as the positions of the joints, the positions of the facial features, etc. The face key points are used to mark the positions of the facial features of the model. The limb key points are used to mark the positions of the limb joints of the model. For example, key point 701 is the face key point of the model in the target template image. Key point 702 is the limb key point of the model in the target template image.

[0119] Among them, the method of detecting skeleton key points includes but is not limited to using a deep convolutional network or a cascaded deep neural network, etc.

[0120] The above bone key points Figure 1 It can be called the first skeleton key point map.

[0121] S507, the key points of the skeleton Figure 1 Remove the facial key points in the image and get the skeleton key points Figure 2 .

[0122] like Figure 7 As shown, the electronic device 100 can Figure 1 The key points of the face (e.g., key point 702, etc.) are removed, and the key points of the limbs (e.g., key point 701, etc.) are retained. Figure 2 The limb key points are included.

[0123] Among them, the method of removing facial key points includes but is not limited to using a deep convolutional network or a method based on face detection. For example, first detect the facial key points of the target template image, and use the smallest area that can contain all the facial key points as the face detection frame. Then, at the skeleton key points, Figure 1 Remove the key points contained in the face detection frame, which is located at the skeleton key points Figure 1 The position in is the same as that in the target template image.

[0124] The above bone key points Figure 2 It can be called the second skeletal key point map.

[0125] S508, using affine transformation matrix to transform key points of human face Figure 1 Perform affine transformation to obtain the transformed face key points Figure 1 , fuse the transformed facial key points in face key points 1 to the skeleton key points Figure 2 In the figure, we get the attitude control diagram.

[0126] like Figure 8As shown, the electronic device 100 can use the affine transformation matrix to transform the key points of the face Figure 1 Perform affine transformation to obtain the transformed face key points Figure 1 The electronic device 100 can transform the key points of the face based on the affine transformation matrix. Figure 1 Perform operations such as shrinking and enlarging to make the key points of the transformed face Figure 1 The position and size of the facial key points in the target template image can replace the position and size of the facial key points in the target template image. The electronic device 100 can fuse the facial key points in the transformed facial key points 1 to the skeleton key points. Figure 2 In the figure, we get the attitude control diagram.

[0127] The key points of the face after the above transformation Figure 1 It can be called the third face key point map.

[0128] S509, using an affine transformation matrix to perform an affine transformation on the personal front face image to obtain a transformed personal front face image, and detecting the edge of the transformed personal front face image to obtain a front face edge map.

[0129] like Fig. 9 As shown, the electronic device 100 can use the affine transformation matrix to perform an affine transformation on the personal frontal face image to obtain the transformed personal frontal face image. The electronic device 100 can perform operations such as shrinking and enlarging the personal frontal face image based on the affine transformation matrix, so that the size and position of the face in the transformed personal frontal face image are the same as the size and position of the model face in the target template image. The electronic device 100 can detect the edge of the face in the transformed personal frontal face image to obtain a frontal face edge map.

[0130] The transformed personal frontal face image may be referred to as a second frontal face image.

[0131] S510: Determine a forehead edge control map from the front face edge map.

[0132] like Fig. 9 As shown, the electronic device 100 can determine the position of the eyebrows of the face in the front face edge map. The electronic device 100 can intercept the edge of the part above the eyebrows of the face from the front face edge map to obtain the forehead edge control map.

[0133] S511 , generating a personalized image through a text-generated graph model based on a first scene template model, a portrait model of a designated user, a posture control graph, a forehead edge control graph, and a first prompt word.

[0134] like Fig.10As shown, the electronic device 100 can use the posture control map, the forehead edge control map, the portrait model of the specified user, and the first scene template model to control the Wensheng map model to generate a personalized image based on the first prompt word.

[0135] In a possible implementation, the electronic device 100 may also use the posture control graph, the portrait model of the designated user, and the first scene template model to control the text graph model to generate a personalized image based on the first prompt word.

[0136] Among them, the Wensheng graph model can generate a personalized image based on the first prompt word. The first prompt word may include a positive prompt word and a negative prompt word. The positive prompt word can guide the Wensheng graph model to generate an image of the style represented by the positive prompt word. For example, the positive prompt words include male (male), short hair (short_hair), black hair (black_hair), so the positive prompt word can guide the Wensheng graph model to generate a personalized image with male, short hair, black hair and other features. The negative prompt word can guide the Wensheng graph model to avoid generating an image of the style represented by the negative prompt word. For example, the negative prompt words include no clothes (noclothes) and rude (nude), so the negative prompt word can guide the Wensheng graph model to generate a personalized image without rude, no clothes and other features.

[0137] Among them, the posture control graph can control the body posture, facial features posture and face shape of the portrait in the personalized image to be consistent with the posture marked by the key points in the posture control graph during the process of generating personalized images using the Wensheng graph model. In this way, the portrait in the personalized image can be accurately controlled, while improving the fidelity of the face shape and facial features position.

[0138] Among them, the forehead edge control map can control the forehead of the portrait in the personalized image to be consistent with the forehead in the forehead edge control map during the process of the Wensheng map model generating the personalized image. In this way, the face shape and facial features of the person can be more closely matched in the personalized image.

[0139] The portrait model of the designated user is used to generate features of the face of the designated user. The portrait model of the designated user can fine-tune the text map model based on the features of the face of the designated user, so that the face in the personalized image generated by the text map model is the same as the face of the designated user in size and position, etc. Therefore, the portrait model of the designated user can make the face of the generated personalized image the face of the designated user.

[0140] The electronic device 100 can adjust the weight of the first scene template model and the weight of the portrait model of the specified user. The greater the weight of the first scene template model, the more obvious the scene features (such as clothing and background, etc.) of the personalized image generated by the Wensheng graph model are. The greater the weight of the portrait model of the specified user, the more obvious the personal features (such as facial features and face shape, etc.) of the personalized image generated by the Wensheng graph model are. Fig.10 As shown in , in the control text graph model, the weight of the first scene template model is set to w1, and the weight of the portrait model of the specified user is set to w2, w1+w2=1. For example, w1 can be 0.5, and w2 can be 0.5. The weight w1 of the first scene template model is the same as the weight w2 of the portrait model of the specified user, so that the generated personalized image can better take into account personal characteristics and scene characteristics.

[0141] Among them, the above-mentioned first scene template model can be a pre-trained model.

[0142] The electronic device 100 can receive user input and determine multiple images of a specified user. The electronic device 100 can detect the gender of the specified user in the multiple images of the specified user. The electronic device 100 can determine a target template image from multiple template images of a specified style based on the user input. The style of the template image in the multiple template images of the specified style can be a ID photo style, a portrait photo style, an artistic photo style, etc. Each template image corresponds to a scene template model and a prompt word in a shooting scene of the template image style. For example, a template image in a portrait photo style corresponds to a scene template model and a prompt word in a shooting scene of a portrait photo style. The target template image corresponds to a first scene template model and a first prompt word in a shooting scene of the target template image style.

[0143] For example, the above Figure 3D to Figure 3G As shown in . The electronic device 100 can receive the user's input and determine multiple images of the designated user. The electronic device 100 can detect that the gender of the designated user in the multiple images of the designated user is male. Based on the user's input, the electronic device 100 can determine that the target template image is a template image 341 from multiple template images of the portrait style. The template image 341 corresponds to a first scene template model and a first prompt word in a shooting scene of the portrait style.

[0144] The above-mentioned electronic device 100 can receive user input and determine multiple images of a specified user. The electronic device 100 can detect the gender of the specified user from the multiple images of the specified user. The electronic device 100 can determine the target template image from multiple template images of a specified style based on the user input. This process can be called the first input.

[0145] The shooting scene of the target template image style may be referred to as a first shooting scene.

[0146] In a possible implementation, the electronic device 100 provides one or more male template images and one or more female template images. The style of the male template image may be a style of a ID photo, a style of a portrait photo, an artistic photo, etc. The style of the female template image may be a style of a ID photo, a style of a portrait photo, an artistic photo, etc. The electronic device 100 may receive a user's selection of a male template image or a female template image, determine a target template image, and simultaneously determine a first scene template model corresponding to the target template image.

[0147] The first scene template model can control the generation of scenes in the personalized image during the process of the text graph model generating the personalized image. For example, the style of the generated personalized image is a portrait photo, and the scene is a business suit and a solid color background.

[0148] The process of training the first scene template model can be as follows:

[0149] The training device first collects several scene-based real images. The training device can annotate label information for each image based on the user's input. The scene-based real image can include any one of a portrait, a ID photo, an artistic portrait, etc. The label information can be male (male), short hair (short_hair), black hair (black_hair), etc. Then, the training device can use the above-mentioned scene-based real images and label information as a scene training set. The training device can use the scene training set to train the first scene template model. Training the first scene template model includes but is not limited to a low-rank adaptation of large language models (LoRA) model based on a large language model with stable diffusion (Stable Diffusion), a DreamBooth model, and the like.

[0150] For example, Fig.11A As shown in the figure, the LoRA of the male image scene is taken as an example. Several male image photos are collected, and label information is manually annotated for each male image photo. The male image photos and the label information of the male image photos are used as the scene training set to train the LoRA model.

[0151] Female image training Fig. 11B As shown in the figure, LoRA of a female image scene is taken as an example. Several female images are collected, and label information is manually annotated for each female image. The female image and the label information of the female image are used as the scene training set to train the LoRA model.

[0152] S512: extracting the facial region of the portrait in the personalized image to obtain a facial mask image.

[0153] like Fig.12 As shown, the electronic device 100 can extract the facial region of the portrait in the personalized image to obtain a facial mask image. The facial mask image is used to represent the expanded facial region of the portrait in the personalized image, and the facial region in the facial mask image is a white region and the non-face region is a black region.

[0154] In one possible implementation, Fig.12 As shown, the electronic device 100 can first segment the facial region of the portrait in the personalized image to obtain a preliminary mask image. The preliminary mask image can be used to characterize the facial region of the portrait in the personalized image. The electronic device 100 expands the preliminary mask image at a certain ratio to obtain a facial mask image.

[0155] The segmentation of the preliminary mask image may use but is not limited to a deep convolutional network or a cascaded deep neural network, and the expansion of the facial mask image may use but is not limited to an image dilation algorithm and a deep learning method.

[0156] The facial mask image is used to represent the expanded facial region of the portrait in the personalized image, and the expanded facial region can be referred to as the first facial region.

[0157] The preliminary mask image may be used to characterize the facial region of the portrait in the personalized image, and the facial region may be referred to as a second facial region.

[0158] S513. Generate a target image through an image-generated image model based on the personalized image, the facial mask image, the first scene template model, the portrait model of the designated user, the posture control map, and the forehead edge control map.

[0159] like Fig.13 As shown, the electronic device 100 can use the facial mask map, the posture control map, the forehead edge control map, the portrait model of the specified user and the first scene template model control map raw image model to finely repair the face of the specified user in the personalized image to obtain the target image.

[0160] In one possible implementation, the electronic device 100 may use the facial mask image, the posture control image, the portrait model of the specified user, and the first scene template model control image model to finely repair the face of the specified user in the personalized image to obtain a target image.

[0161] The image-generating model can generate a target image based on an input image. The input image can be the above Fig.10The personalized image shown. The graph-to-graph model can generate a target image based on the personalized image. In this way, the randomness of the target image generation can be reduced, and the content in the target image is generated toward the content in the personalized image, thereby obtaining the desired target image.

[0162] Among them, the facial mask image can help the image generation model to repair (inpainting) the facial area of ​​the personalized image during the process of the image generation model generating the target image. Fig.12 As shown in , the white area in the facial mask image is used to represent the area where the facial area of ​​the portrait is expanded in the personalized image. The image generation model can determine the area of ​​the face in the personalized image through the facial mask image, and then repair the personalized image in this area so that the face of the target image is the same size as the face of the specified user. In this way, the facial area of ​​the portrait can be refined, while the clothing and background remain unchanged, improving the fidelity of the portrait.

[0163] The first scene template model may be a pre-trained model. Fig.10 The first scene template model can control the generation of scenes in the target image during the process of the image generation model generating the target image. For example, the style of the generated target image is a portrait photo, and the scene is a business suit and a solid color background.

[0164] Among them, the posture control graph can control the body posture, facial features and face shape of the portrait in the target image to be consistent with the posture marked by the key points in the posture control graph during the process of generating the target image by the image-generated model. In this way, the portrait in the target image can be accurately controlled, and the fidelity of the face shape and facial features position can be improved.

[0165] Among them, the forehead edge control map can control the forehead of the portrait in the target image to be consistent with the forehead in the forehead edge control map during the process of the image-generated image model generating the target image. In this way, the face shape and facial features of the person can be more closely matched in the target image.

[0166] The portrait model of the designated user is used to generate the features of the face of the designated user. The portrait model of the designated user can fine-tune the image generation model based on the features of the face of the designated user, so that the face in the target image generated by the image generation model is the same as the face of the designated user in size and position, etc. Therefore, the portrait model of the designated user can make the face of the generated target image the face of the designated user.

[0167] The electronic device 100 can adjust the weight of the first scene template model and the weight of the portrait model of the specified user. The greater the weight of the first scene template model, the more obvious the scene features (such as clothing and background, etc.) of the target image generated by the image generation model. The greater the weight of the portrait model of the specified user, the more obvious the personal features (such as facial features and face shape, etc.) of the target image generated by the image generation model. Fig.13 As shown, in the control graph generation model, the weight of the first scene template model is set to w3, and the weight of the portrait model of the specified user is set to w4, w3+w4=1. The weight w3 can be less than the above Fig.10 The weight w1 shown in , for example, w1 can be 0.5, w3 can be 0.15. The reduction of the weight of the first scene template model can retain some scene features (such as background and clothing, etc.) in the target image. The weight w4 is greater than the above Fig.10 The weight w2 shown in . For example, w4 can be 0.85. The increase in the weight of the portrait model of the specified user can improve the fidelity of the face part in the generated target image. In this way, the generated portrait photo can meet both the scene requirements and the high fidelity requirements.

[0168] In a possible implementation, the electronic device 100 can use the facial mask image, the posture control image, the portrait model of the specified user, and the first scene template model control image model to finely repair the face of the specified user in the personalized image based on the first prompt word to obtain a target image. The first prompt word can refer to the above Fig.10 The first prompt word shown in is not repeated here.

[0169] In a possible implementation, the electronic device 100 can use the facial mask map, the posture control map, the forehead edge control map, the portrait model of the specified user, and the first scene template model control map raw image model to finely repair the face of the specified user in the personalized image based on the first prompt word to obtain a target image. The first prompt word can refer to the above Fig.10 The first prompt word shown in is not repeated here.

[0170] First, a personalized image is generated through the text-generated graph model, and then a target image is generated based on the personalized image through the graph-generated graph model, which improves the fidelity of the facial contour and fine features of the facial features, and increases the similarity between the face of the generated target image and the face of the designated user.

[0171] In a possible implementation, the electronic device 100 may execute S501 to S511 above and display a personalized image, such as the above Figure 3G The personalized image shown in the Figure 4F The personalized image shown is portrait photo 451.

[0172] For example, the electronic device 100 obtains a personal image training set and a target template image. The electronic device 100 sends the obtained personal image training set and the target template image to the electronic device 100. Then, the electronic device 100 trains the portrait model of the specified user through the personal image training set. The electronic device 100 selects a personal frontal face image from the personal image training set, and detects the key points of the face in the personal frontal face image to obtain the face key points. Figure 1 The electronic device 100 detects the key points of the face in the target template image and obtains the key points of the face. Figure 2 The electronic device 100 determines the key points of the face Figure 1 And facial key points Figure 2 The electronic device 100 detects the skeleton key points in the target template image and obtains the skeleton key points Figure 1 The electronic device 100 converts the skeleton key points Figure 1 Remove the facial key points in the image and get the skeleton key points Figure 2 The electronic device 100 uses an affine transformation matrix to transform key points of the face Figure 1 Perform affine transformation to obtain the transformed face key points Figure 1 , fuse the transformed facial key points in face key points 1 to the skeleton key points Figure 2 , and obtain the posture control map. The electronic device 100 uses the affine transformation matrix to perform an affine transformation on the personal front face image to obtain the transformed personal front face image, and detects the edge of the transformed personal front face image to obtain the front face edge map. The electronic device 100 determines the forehead edge control map from the front face edge map. The electronic device 100 generates a personalized image based on the first scene template model, the portrait model of the specified user, the posture control map, the forehead edge control map and the first prompt word through the Wensheng graph model. The electronic device 100 can display personalized images, such as the above Figure 3G The personalized image shown in the Figure 4F The personalized image shown is portrait photo 451.

[0173] In a possible implementation, the electronic device 100 may execute the above S501 and S504, and the server 200 may execute the above S502 to S513. The server 200 may return the target image to the electronic device 100, and the electronic device 100 may display the target image, such as the above Figure 3G The target image displayed in the image is a photo 351 or Figure 4F The target image shown in is image 451.

[0174] For example, the electronic device 100 obtains a personal image training set and a target template image. The electronic device 100 sends the obtained personal image training set and the target template image to the server 200. Then, the server 200 trains the portrait model of the specified user through the personal image training set. The server 200 selects a personal frontal face image from the personal image training set, detects the key points of the face in the personal frontal face image, and obtains the face key points. Figure 1 The server 200 detects the key points of the face in the target template image and obtains the key points of the face Figure 2 Server 200 determines key points of the face Figure 1 And facial key points Figure 2 The server 200 detects the skeleton key points in the target template image and obtains the skeleton key points. Figure 1 The server 200 converts the skeleton key points Figure 1 Remove the facial key points in the image and get the skeleton key points Figure 2 The server 200 uses an affine transformation matrix to transform key points of the face Figure 1 Perform affine transformation to obtain the transformed face key points Figure 1 , fuse the transformed facial key points in face key points 1 to the skeleton key points Figure 2 , and obtain a posture control map. The server 200 uses an affine transformation matrix to perform an affine transformation on the personal front face image to obtain a transformed personal front face image, and detects the edge of the transformed personal front face image to obtain a front face edge map. The server 200 determines a forehead edge control map from the front face edge map. The server 200 generates a personalized image based on the first scene template model, the portrait model of the specified user, the posture control map, the forehead edge control map and the first prompt word through the text-generated graph model. The server 200 extracts the facial area of ​​the portrait in the personalized image to obtain a facial mask map. The server 200 generates a target image based on the personalized image, the facial mask map, the first scene template model, the portrait model of the specified user, the posture control map and the forehead edge control map through the graph-generated graph model. The server 200 can return the target image to the electronic device 100.

[0175] In a possible implementation, the electronic device 100 may execute the above S501 and S504, and the server 200 may execute the above S502 to S511. The server 200 may return the personalized image to the electronic device 100, and the electronic device 100 may display the personalized image, such as the above Figure 3G The personalized image shown in the Figure 4F The personalized image shown is portrait photo 451.

[0176] For example, the electronic device 100 obtains a personal image training set and a target template image. The server 200 sends the obtained personal image training set and the target template image to the server 200. Then, the server 200 trains the portrait model of the specified user through the personal image training set. The server 200 selects a personal frontal face image from the personal image training set, detects the key points of the face in the personal frontal face image, and obtains the key points of the face. Figure 1 The server 200 detects the key points of the face in the target template image and obtains the key points of the face Figure 2 Server 200 determines key points of the face Figure 1 And facial key points Figure 2 The server 200 detects the skeleton key points in the target template image and obtains the skeleton key points. Figure 1 The server 200 converts the skeleton key points Figure 1 Remove the facial key points in the image and get the skeleton key points Figure 2 The server 200 uses an affine transformation matrix to transform key points of the face Figure 1 Perform affine transformation to obtain the transformed face key points Figure 1 , fuse the transformed facial key points in face key points 1 to the skeleton key points Figure 2 , and obtain the posture control map. The server 200 uses the affine transformation matrix to perform an affine transformation on the personal front face image to obtain the transformed personal front face image, and detects the edge of the transformed personal front face image to obtain the front face edge map. The server 200 determines the forehead edge control map from the front face edge map. The server 200 generates a personalized image based on the first scene template model, the portrait model of the specified user, the posture control map, the forehead edge control map and the first prompt word through the Wensheng graph model. The server 200 can return the personalized image to the electronic device 100. The electronic device 100 can display the personalized image, such as the above Figure 3G The personalized image shown in the Figure 4F The personalized image shown is portrait photo 451.

[0177] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0178] The embodiments of the present application also provide a computer program product. When the computer program product is executed on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments.

[0179] The embodiment of the present application also provides a chip system, the chip system includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the steps of any method embodiment of the present application. The chip system can be a single chip or a chip module composed of multiple chips.

[0180] The term "user interface (UI)" in the specification and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the electronic device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as <head> and <body>. A node corresponds to a control or attribute in the interface. After being parsed and rendered, the node is presented as user-visible content. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code. For example, HTML is defined by <body> tags or nodes in the web page source code. 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0181] The most common form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operation that is displayed in a graphical manner. It can be an icon, window, control or other interface element displayed on the display screen of an electronic device, where a control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.

[0182] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0183] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0184] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. An image generation method, characterized in that: include: receiving a first input; In response to the first input, determining a first scene template model and a first prompt word in a first shooting scene; Acquire a personal image training set, wherein the personal image training set includes a plurality of personal images of a specified user; Training a first portrait model using the personal image training set, where the first portrait model is used to generate facial features of the designated user; A first frontal face image is obtained from the personal image training set, and a first facial key point map is generated based on the first frontal face image; the first frontal face image includes a face of a specified user in a frontal face posture, and the first facial key point map includes facial key points in the first frontal face image; Acquire a target template image, and generate a first skeleton key point map based on the target template image; the target template image includes a face of a model posing in a specified posture, and the first skeleton key point map includes face key points and limb key points in the target template image; Replacing the facial key points in the first skeleton key point map with the facial key points in the first facial key point map to obtain a posture control map; The posture control graph, the first portrait model and the first scene template model are used to control a Vincent graph model to generate a personalized image based on the first prompt word, wherein the personalized image includes a face of the designated user when posing the designated posture in the first shooting scene.

2. The method according to claim 1, characterized in that The method further comprises: Extracting a facial region of a person in the personalized image to obtain a facial mask image, wherein the facial mask image is used to characterize a first facial region of the person in the personalized image; The face of the designated user in the personalized image is finely repaired using the facial mask image, the posture control image, the first portrait model and the first scene template model control image model to obtain a target image.

3. The method according to claim 2, characterized in that The similarity between the face in the target image and the face in the first frontal face image is greater than the similarity between the face in the personalized image and the face in the first frontal face image.

4. The method according to claim 2 or 3, characterized in that: The method further comprises: Setting a model weight of the first portrait model in a process in which the Wensheng graph model generates the personalized image to a first weight; The model weight of the first scene template model in the process of the cultural graph model generating the personalized image is set as a second weight.

5. The method according to claim 4, characterized in that The method further comprises: Setting the model weight of the first portrait model in the process of the image generation model generating the target image to a third weight; The model weight of the first scene template model in the process of the image generation model generating the target image is set to a fourth weight, and the fourth weight is less than the third weight.

6. The method according to claim 5, characterized in that The first weight is smaller than the third weight, and the second weight is larger than the fourth weight.

7. The method according to claim 6, characterized in that The first weight is equal to the second weight.

8. The method according to any one of claims 2 to 7, characterized in that: The method further comprises: Using the affine transformation matrix to perform an affine transformation on the first frontal face image to obtain a second frontal face image, wherein the size and position of the face in the second frontal face image are the same as the size and position of the model's face in the target template image; Detecting the edge of the face in the second frontal face image to obtain a frontal face edge map; Determine the position of the eyebrows in the front face edge map; Cutting the edge of the portion above the eyebrows from the front face edge map to obtain a forehead edge control map; The step of using the posture control graph, the first portrait model, and the first scene template model to control the Wensheng graph model to generate a personalized image based on the first prompt word specifically includes: The posture control map, the forehead edge control map, the first portrait model and the first scene template model are used to control a Vincent graph model to generate a personalized image based on the first prompt word.

9. The method according to claim 8, characterized in that The step of using the facial mask image, the posture control image, the first portrait model, and the first scene template model control image model to finely repair the face of the designated user in the personalized image to obtain a target image specifically includes: The face of the designated user in the personalized image is finely repaired using the facial mask image, the forehead edge control image, the posture control image, the first portrait model and the first scene template model control image model to obtain a target image.

10. The method according to any one of claims 2 to 9, characterized in that: The step of extracting the facial region of the portrait in the personalized image to obtain a facial mask image specifically includes: Segmenting a facial region of the portrait in the personalized image to obtain a preliminary mask image, wherein the preliminary mask image can be used to characterize a second facial region of the portrait in the personalized image; The preliminary mask image is expanded according to a specified ratio to obtain the facial mask image.

11. The method according to any one of claims 1 to 10, characterized in that The step of training the first portrait model by using the personal image training set specifically includes: Resizing the images in the personal image training set to a specified size; The portrait model of the designated user is trained using the images in the personal image training set that are adjusted to the designated size.

12. The method according to any one of claims 1 to 11, characterized in that The step of replacing the facial key points in the first skeleton key point map with the facial key points in the first facial key point map to obtain a posture control map specifically includes: Acquire a target template image, and generate a second facial key point map based on the target template image, wherein the second facial key point map includes facial key points in the target template image; Determine an affine transformation matrix between the first facial key point map and the second facial key point map, wherein the affine transformation matrix is ​​used to characterize a transformation relationship between sizes and positions of facial key points in the first facial key point map and facial key points in the second facial key point map; Performing an affine transformation on the first facial key point map using the affine transformation matrix to obtain a third facial key point map; Remove the face key points in the first skeleton key point map, retain the limb key points, and obtain a second skeleton key point map; The facial key points in the third facial key point map are fused into the second skeleton key point map to obtain the posture control map.

13. An electronic device, characterized in that: The method comprises one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store a computer program, and when the one or more processors execute the computer program, the method as described in any one of claims 1 to 12 is executed.

14. An electronic device, characterized in that: The method comprises one or more functional modules, wherein the one or more functional modules are used to execute the method according to any one of claims 1 to 12.

15. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes a processing circuit and an interface circuit, the interface circuit is used to receive instructions and transmit them to the processing circuit, and the processing circuit is used to run the instructions to execute the method as described in any one of claims 1-12.

16. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a processor of an electronic device, cause the method according to any one of claims 1 to 12 to be executed.

Images