A training and image generation method and device for a gender conversion model of a person image
The method enhances gender conversion flexibility in image processing by training models with unpaired image sets and adjusting irregular features, resulting in realistic gender-converting models for diverse image applications.
Patent Information
- Application Number
- CN202010796905.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-10
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-08-10
AI Technical Summary
Existing image processing technologies face poor flexibility in gender conversion of person images, particularly in generating 3D virtual or cartoon images, leading to suboptimal results.
A method involving unpaired image sample sets of male and female characters, trained using recurrent adversarial generation networks and paired adversarial generative networks to create gender-converting models, with optional adjustments for irregular features like hair and clothing, and data cleaning to enhance realism.
The solution provides flexible and realistic gender-converting models, enabling the generation of paired image data with opposite genders, suitable for generating rich magic expressions.
Smart Images

Figure CN114078082B_ABST
Abstract
Description
Technical Field
[0001] The disclosed embodiments relate to image processing technology, and more particularly to a training and image generation method and device for a gender conversion model of a person image. Background Art
[0002] In the related art, the facial state of a person in an image can be recognized, and then the facial state of the person in the image can be converted into other images, such as 3D (3-Dimensional) virtual images, cartoon images, or characters in fairy tales, etc. However, this method has poor flexibility when performing gender conversion on a person image. Summary of the invention
[0003] The present disclosure provides a training and image generation method and device for a gender conversion model of a person image, so as to at least solve the problem of poor flexibility in gender conversion of a person image in the related art.
[0004] The technical solution of the embodiment of the present disclosure is as follows:
[0005] According to a first aspect of an embodiment of the present disclosure, a method for training a gender conversion model for a person image is provided, the method comprising:
[0006] Acquire an unpaired image sample set, wherein the unpaired image sample set includes a male character image set and a female character image set that have no pairing relationship; the pairing relationship represents: a correspondence between another character image obtained by changing the gender of any character image and keeping the background image unchanged and any of the character images;
[0007] Inputting the unpaired image sample set into a first deep learning model for training to obtain a paired image generation model, wherein the first deep learning model is a recurrent adversarial generation network, and the paired image generation model is used to generate a person image having the paired relationship for each person image in the unpaired image sample set;
[0008] Inputting the image of the person to be paired into the paired image generation model, and outputting the target person image; wherein the image of the person to be paired and the target person image have a paired relationship, and constitute a paired image sample;
[0009] The paired image sample set is input into a second deep learning model for training to obtain a character gender conversion model, wherein the second deep learning model is a paired adversarial generative network, and the character gender conversion model is used to perform a character gender conversion operation.
[0010] Optionally, before the step of inputting the paired image sample set into the person image gender conversion model for training, the method further includes:
[0011] According to the gender of the person in the target person image, irregular image feature adjustment is performed to conform to the gender characteristics of the person; wherein the irregular image feature includes at least one of the following: the hair of the person in the target person image, the clothes worn by the person, or the accessories worn by the person.
[0012] Optionally, the step of adjusting the irregular image features in accordance with the gender of the person in the target person image includes:
[0013] Identify the gender, head outline and posture of the target person in the image;
[0014] According to the gender of the character, the head outline of the character and the posture of the character, the replacement hairstyle is updated to the target character image to perform irregular image feature adjustment.
[0015] Optionally, after updating the replacement hairstyle to the target person image to perform the irregular image feature adjustment step, the method further includes:
[0016] Identifying exposed hair of a person in the target person image, wherein the exposed hair of the person is the hair of the person outside the replacement hairstyle;
[0017] When the exposed hair covers the human face area, determining the skin color of the area covered by the exposed hair according to the skin color of the human face not covered by the exposed hair, so as to eliminate the exposed hair;
[0018] and / or,
[0019] When the exposed hair does not cover the human face area, the exposed hair is eliminated according to the image background not covered by the exposed hair.
[0020] Optionally: the person image to be paired includes at least one image in the unpaired image sample set.
[0021] Optionally, before inputting the unpaired image sample set into the paired image generation model for training, the method further includes:
[0022] Performing data cleaning on the unpaired image sample set to filter out abnormal person images in the unpaired image sample set, wherein the abnormal person images include at least one of the following images: an image that does not contain facial features, or an overexposed image;
[0023] and / or,
[0024] The size of each person image in the unpaired image sample set is normalized to make the sizes of each person image the same.
[0025] According to a second aspect of an embodiment of the present disclosure, there is provided an image generating method, the method comprising:
[0026] Obtain at least one original character image;
[0027] An original character image is input into a character image gender conversion model, and a target character image having a pairing relationship with the original character image is output, wherein the character image gender conversion model is trained according to the training method of the character image gender conversion model according to any one of claims 1-6; the pairing relationship represents: the correspondence between another character image obtained by changing the gender of any character image while keeping the background image unchanged and any of the character images.
[0028] Optionally, the step of obtaining at least one original character image includes:
[0029] At least one key frame is extracted from the continuous video frames provided by the user through the client according to a set rule, and the key frame is used as the original character image.
[0030] According to a third aspect of an embodiment of the present disclosure, a training device for a character image gender conversion model is provided, the device comprising:
[0031] The acquisition module is configured to acquire an unpaired image sample set, wherein the unpaired image sample set includes a male character image set and a female character image set that have no pairing relationship; the pairing relationship represents: a correspondence between another character image obtained by changing the gender of any character image and keeping the background image unchanged and any of the character images;
[0032] A training module, inputting the unpaired image sample set into a first deep learning model for training to obtain a paired image generation model, wherein the paired image generation model is a recurrent adversarial generation network, and the paired image generation model is used to generate a person image having the paired relationship for each person image in the unpaired image sample set;
[0033] An output module is configured to input the image of the person to be paired into the paired image generation model and output a target person image; wherein the image of the person to be paired and the target person image have a paired relationship and constitute a paired image sample;
[0034] The training module is configured to input the paired image sample set into the second deep learning model for training to obtain a character gender conversion model, wherein the second deep learning model is a paired adversarial generative network, and the character gender conversion model is used to perform a character gender conversion operation.
[0035] Optionally, the device further comprises:
[0036] The adjustment module is configured to adjust the irregular image features in accordance with the gender of the person in the target person image; wherein the irregular image features include at least one of the following: the hair of the person in the target person image, the clothes worn by the person, or the accessories worn by the person.
[0037] Optionally, the adjustment module is specifically configured to identify the gender, head outline and posture of the person in the target person image;
[0038] According to the gender of the character, the head outline of the character and the posture of the character, the replacement hairstyle is updated to the target character image to perform irregular image feature adjustment.
[0039] Optionally, the device further comprises:
[0040] A recognition module, configured to recognize exposed hair of a person in the target person image, wherein the exposed hair of the person is the hair of the person outside the replacement hairstyle;
[0041] When the exposed hair covers the human face area, determining the skin color of the area covered by the exposed hair according to the skin color of the human face not covered by the exposed hair, so as to eliminate the exposed hair;
[0042] and / or,
[0043] When the exposed hair does not cover the human face area, the exposed hair is eliminated according to semantic information of the image background not covered by the exposed hair.
[0044] Optionally, the device further comprises:
[0045] A cleaning module is configured to perform data cleaning on the unpaired image sample set to filter out abnormal person images in the unpaired image sample set, wherein the abnormal person images include at least one of the following images: an image that does not contain facial features, or an overexposed image;
[0046] and / or,
[0047] The size of each person image in the unpaired image sample set is normalized to make the sizes of each person image the same.
[0048] According to a fourth aspect of an embodiment of the present disclosure, there is provided an image generating device, the device comprising:
[0049] An acquisition module, configured to acquire at least one original character image;
[0050] The output module is configured to input an original character image into a character image gender conversion model, and output a target character image having a pairing relationship with the original character image, wherein the character image gender conversion model is trained according to the character image gender conversion model training method described in the first aspect of the present disclosure; the pairing relationship represents: the correspondence between another character image obtained by changing the gender of any character image while keeping the background image unchanged and any of the character images.
[0051] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, characterized in that it includes:
[0052] processor;
[0053] a memory for storing instructions executable by the processor;
[0054] The processor is configured to execute the command to implement a training method for a character image gender conversion model as described in any embodiment of the present disclosure, or an object image generation method as described in any embodiment of the present disclosure.
[0055] According to the sixth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the character image gender conversion model or the image generation method described in any embodiment of the present disclosure.
[0056] According to an eighth aspect of the embodiments of the present disclosure, a computer program product is provided. When the instructions in the computer program product are executed by a processor of an electronic device, the training method of the paired image data generation model or the image generation method described in any embodiment of the present disclosure is implemented.
[0057] The technical solution provided by the embodiment of the present disclosure brings at least the following beneficial effects: by obtaining an unpaired image sample set, the unpaired image sample set includes a male character image set and a female character image set that have no pairing relationship; the unpaired image sample set is input into the first deep learning model for training to obtain a paired image generation model; the image of the character to be paired is input into the paired image generation model to output the target character image; the paired image sample set is input into the second deep learning model for training to obtain a character gender conversion model, which is used to perform the character gender conversion operation. The solution of the present disclosure can solve the problem of poor flexibility in gender conversion of character images in the related art, and can obtain paired image data with the opposite gender to the character image, providing a basis for the subsequent generation of rich magic expressions; it can solve the problem of poor flexibility in gender conversion of character images in the related art, and can obtain paired image data with the opposite gender to the character image, providing a basis for the subsequent generation of rich magic expressions.
[0058] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0060] Figure 1 The present invention is a flowchart of a method for training a gender conversion model for a person image according to an exemplary embodiment.
[0061] Figure 2 The present invention is a flowchart of a method for training a gender conversion model for a person image according to an exemplary embodiment.
[0062] Figure 3 The present invention is a flowchart of a method for training a gender conversion model for a person image according to an exemplary embodiment.
[0063] Figure 4 The figure is a flowchart of an image generating method according to an exemplary embodiment.
[0064] Figure 5 The figure is a flowchart of an image generating method according to an exemplary embodiment.
[0065] Figure 6 The present invention is a flowchart of a method for training a gender conversion model for a person image according to an exemplary embodiment.
[0066] Figure 7a The figure is a flowchart of an image generating method according to an exemplary embodiment.
[0067] Figure 7b The figure is a schematic diagram of a magic expression generated by a method for generating a magic expression character image according to an exemplary embodiment.
[0068] Figure 7c The figure is a schematic diagram of a magic expression generated by an image generation method according to an exemplary embodiment.
[0069] Figure 7d The figure is a schematic diagram of a magic expression generated by an image generation method according to an exemplary embodiment.
[0070] Figure 8 The present invention is a block diagram of a training device for a character image gender conversion model according to an exemplary embodiment.
[0071] Fig. 9The figure is a block diagram of an image generating device according to an exemplary embodiment.
[0072] Fig.10 The present invention is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION
[0073] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0074] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0075] Figure 1 is a flowchart of a method for training a character image gender conversion model according to an exemplary embodiment. Figure 1 As shown, the training method of the character image gender conversion model can be executed by a training device for the character image gender conversion model, which can be implemented by software and / or hardware and used in an electronic device, which can be a server. The method includes the following steps.
[0076] In step S11, an unpaired image sample set is obtained, wherein the unpaired image sample set includes a male character image set and a female character image set that have no pairing relationship.
[0077] The pairing relationship involved in this embodiment means: the correspondence between any character image and another character image obtained by changing the gender of any character image and keeping the background image unchanged. For example, male character image A and female character image A have a pairing relationship; then, except for the opposite gender, the male character image A and the female character image A have the same features, for example, the backgrounds of the two character images and the facial features of the characters in the two character images are the same.
[0078] In this embodiment, the unpaired image sample set may include a male character image set and a female character image set; wherein, there is no pairing relationship between the multiple male character images included in the male character image set and the multiple female character images included in the female character image set, that is, there is no pairing relationship between the multiple character images included in the unpaired image sample set.
[0079] In an optional implementation of the present embodiment, the acquired unpaired image sample set may include 20,000 person images, wherein the 20,000 person images include 10,000 male person images and 10,000 female person images, and there is no pairing relationship between each male person image and each female person image.
[0080] In step S12, the unpaired image sample set is input into the first deep learning model for training to obtain a paired image generation model.
[0081] Among them, the first deep learning model is a recurrent adversarial generation network; the paired image generation model is used to generate character images with pairing relationships for each character image in the unpaired image sample set.
[0082] In an optional implementation of this embodiment, after obtaining the unpaired image sample set, the unpaired image sample set can be input into the first deep learning model for training, so as to adjust the parameters of the first deep learning model. When the number of iterations reaches a set number (for example, 100,000 times or 200,000 times, which is not limited in this embodiment), or when the first deep learning model converges, the training of the first deep learning model can be stopped. Among them, the first deep learning model is a cycle-constraint adversarial network (Cycle GAN), so as to obtain a paired image generation model. It should be noted that the paired image generation model can generate a paired image for each character image in the unpaired image sample set, that is, it can generate a character image that has a paired relationship with each character image in the unpaired image sample set.
[0083] Exemplarily, after obtaining the unpaired image sample set, the unpaired image sample set can be input into the Cycle GAN network for training, so as to adjust the network parameters of the Cycle GAN network. When the number of iterations reaches a set number (for example, 100,000 times or 200,000 times), or when the Cycle GAN network converges, the training of the Cycle GAN network is stopped to obtain a paired image generation model.
[0084] In step S13, the image of the person to be paired is input into the pairing image generation model, and the target person image is output.
[0085] Among them, the image of the person to be paired and the image of the target person have a pairing relationship, constituting a paired image sample.
[0086] It should be noted that the person image to be paired involved in this embodiment can be any image in the unpaired image sample set obtained in step S11; at the same time, this embodiment does not limit the gender of the person in the image to be paired, and it can be either a male person image or a female person image.
[0087] In an optional implementation of the present embodiment, after inputting the image of the person to be paired into the trained paired image generation model, a target person image paired with the image of the person to be paired can be obtained, wherein the gender of the image of the person to be paired is opposite to that of the target person image, and the background image is the same; each image of the person to be paired and the target person image it is paired with constitute a paired image sample, that is, they have a paired relationship. Exemplarily, if the image of the person to be paired is a male person image A, inputting it into the paired image generation model can obtain a female person image A having a paired relationship with the male person image A, and the male person image A and the female person image A constitute a paired image sample; if the image of the person to be paired is a female person image B, inputting it into the paired image generation model can obtain a male person image B having a paired relationship with the female person image B, and the female person image B and the male person image B constitute a paired image sample.
[0088] In step S14, the paired image sample set is input into the second deep learning model for training to obtain a character image gender conversion model.
[0089] Among them, the second deep learning model is a paired adversarial generative network, and the character gender conversion model is used to perform character gender conversion operations. It should be noted that the character gender conversion model trained in this embodiment can convert the gender of any character image, and can keep the background image of the character image unchanged. For example, by inputting a male character image A into the character gender conversion model trained in this embodiment, a female character image A can be obtained; wherein the male character image A is of opposite gender to the female character image, and the background image is the same.
[0090] It should be noted that the paired image sample set involved in this embodiment may include multiple paired image samples, for example, 100, 10,000 or 100,000, etc., which is not limited in this embodiment.
[0091] In an optional implementation of this embodiment, after obtaining the paired image sample set, the obtained paired sample set can be further input into the second deep learning model for training, so as to optimize the various parameters in the second deep learning model. When the number of iterations reaches a set number (for example, 100,000 times or 200,000 times, which is not limited in this embodiment), or when the second deep learning model converges, the training of the second deep learning model can be stopped to obtain a character image gender conversion model; wherein the second deep learning model is a paired adversarial generative network (Pix2Pix).
[0092] Exemplarily, after obtaining a paired image sample set, the paired image sample set can be input into a Pix2Pix network for training, thereby adjusting the network parameters of Pix2Pix. When the number of iterations reaches a set number (for example, 100,000 or 200,000 times), or when the Pix2Pix network converges, the training of the Pix2Pix network is stopped, thereby obtaining a character image gender conversion model.
[0093] The solution of this embodiment obtains an unpaired image sample set, which includes a male character image set and a female character image set that have no pairing relationship; inputs the unpaired image sample set into a first deep learning model for training to obtain a paired image generation model; inputs the image of the character to be paired into the paired image generation model to output the target character image; inputs the paired image sample set into a second deep learning model for training to obtain a character gender conversion model; this can solve the problem of poor flexibility in gender conversion of character images in related technologies, and can obtain paired image data of the opposite gender to the character image, providing a basis for subsequent generation of rich magic expressions.
[0094] Figure 2 1 is a flowchart of a training method for a character image gender conversion model according to an exemplary embodiment. This embodiment is a further refinement of the above technical solution. The technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 2 As shown, the training method of the character image gender conversion model includes the following steps.
[0095] In step S21, an unpaired image sample set is obtained, wherein the unpaired image sample set includes a male character image set and a female character image set that have no pairing relationship.
[0096] In step S22, the unpaired image sample set is input into the first deep learning model for training to obtain a paired image generation model.
[0097] In step S23, the image of the person to be paired is input into the pairing image generation model, and the target person image is output.
[0098] In step S24, according to the gender of the person in the target person image, irregular image feature adjustment is performed to conform to the gender characteristics of the person.
[0099] The irregular image feature may include at least one of the following: the hair of the target person in the image, the clothes worn by the person, or the accessories worn by the person. The accessories worn by the person may include earrings, necklaces, or hairpins, which are not limited in this embodiment.
[0100] In an optional implementation of the present embodiment, after the image of the person to be paired is input into the pairing image generation model to generate a target person image to be paired with the image of the person to be paired, non-regular image feature adjustments that conform to the gender characteristics of the person can be further performed based on the gender of the person in the target person image. For example, the hairstyle of the person or the clothes worn by the person can be changed based on the gender of the person in the target person image.
[0101] The advantage of such a setting is that by adjusting the irregular features of the character's gender characteristics, the realism of the target image after gender conversion can be enhanced.
[0102] Optionally, based on the gender of the person in the target person image, performing irregular image feature adjustment that conforms to the gender characteristics of the person may include: identifying the gender of the person, the head outline of the person, and the posture of the person in the target person image; based on the gender of the person, the head outline of the person, and the posture of the person, updating the replacement hairstyle to the target person image to perform irregular image feature adjustment; wherein the replacement hairstyle may be stored in a hairstyle library.
[0103] In an optional implementation of this embodiment, the gender, head outline and posture of the person in the target person image can be first identified, and the replacement hairstyle can be further updated to the target person image based on the identified gender, head outline and posture of the person, wherein the replacement hairstyle can be any hairstyle stored in the hairstyle library, for example, long straight hair, black curly hair or short blond hair, etc., which is not limited in this embodiment.
[0104] In another optional implementation of this embodiment, after identifying the gender, head outline and posture of the person in the target person image, a camera posture estimation algorithm based on PNP (Perspective-n-Point, multi-point perspective imaging) can be further used to update the replacement hairstyle to the target person image, for example, updating the replacement hairstyle on the head of the person in the target person image.
[0105] Optionally, after updating the replacement hairstyle to the target person image to perform irregular image feature adjustment, the method may also include: identifying the exposed hair of the person in the target person image, wherein the exposed hair of the person is the hair of the person outside the replacement hairstyle; if it is determined that the exposed hair covers the face area, determining the skin color of the area covered by the exposed hair based on the skin color of the face not covered by the exposed hair to eliminate the exposed hair; and / or, if it is determined that the exposed hair does not cover the face area, eliminating the exposed hair based on semantic information of the image background not covered by the exposed hair.
[0106] In an optional implementation of this embodiment, after the replacement hairstyle is updated to the target person image, the exposed hair of the person in the target person image can be further identified, wherein the exposed hair of the person can be the hair of the person outside the replacement hairstyle. For example, if the hairstyle of the target person image before the replacement hairstyle is updated is long hair with bangs, and the replacement hairstyle is short hair without bangs, then the exposed hair is the bangs not covered by the short hair without bangs, and the long hair under the ears.
[0107] Further, if it is determined that the exposed hair covers an area of the face (e.g., the forehead), the skin color of the area of the face covered by the exposed hair can be determined based on the skin color of the face not covered by the exposed hair (e.g., the cheek skin color), thereby reconstructing the skin color of the face in the area covered by the bangs to eliminate the exposed hair covering the face area.
[0108] For example, if it is determined that the exposed hair covers the forehead, the forehead skin can be masked (e.g., painted white), and the target person image is used for supervision to reconstruct the skin of the masked area. For example, the process of removing bangs is to cover the user's bangs area, and after calculation by the bangs removal network, the masked area is reconstructed as the user's skin, and the bangs removal is completed.
[0109] Further, if it is determined that the exposed hair does not cover the face area, the exposed hair is eliminated according to the semantic information of the image background not covered by the exposed hair. For example, if it is determined that the exposed hair covers the shoulders of the person, the exposed hair can be eliminated according to the semantic information of the image background not covered by the exposed hair.
[0110] In an optional implementation of this embodiment, the Inpainting algorithm can be used to remove exposed hair that is not covered in the face area and restore semantic information such as the corresponding background.
[0111] The advantage of such a setting is that the user's hair exposed outside the replacement hairstyle can be eliminated, thereby enhancing the realism of the target image after the gender is changed.
[0112] In step S25, the paired image sample set is input into the second deep learning model for training to obtain a character gender conversion model.
[0113] The solution of this embodiment, after obtaining the target person image, can identify the person's gender, person's head outline and person's posture in the target person image; according to the person's gender, person's head outline and person's posture, the replacement hairstyle is updated to the target person image to adjust the irregular image features, and while obtaining the paired image data of the opposite gender to the person image, the irregular image features of the person's gender features can also be adjusted, thereby enhancing the realism of the target image after the gender conversion, and providing a basis for the subsequent generation of rich magic expressions.
[0114] Figure 3 1 is a flowchart of a training method for a character image gender conversion model according to an exemplary embodiment. This embodiment is a further refinement of the above technical solution. The technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 3 As shown, the training method of the character image gender conversion model includes the following steps.
[0115] In step S31, an unpaired image sample set is obtained, wherein the unpaired image sample set includes a male character image set and a female character image set that have no pairing relationship.
[0116] In step S32, data cleaning is performed on the unpaired image sample set to filter out abnormal person images in the unpaired image sample set; and / or the person images in the unpaired image sample set are size normalized to make the sizes of the person images the same.
[0117] In an optional implementation of the present embodiment, after obtaining the unpaired image sample set, the obtained unpaired image sample set may be further cleaned. For example, unclear image samples may be filtered out, image samples that do not contain facial features may be filtered out, or overexposed image samples may be filtered out, etc. This is not limited in the present embodiment.
[0118] Furthermore, the size of each person image in the unpaired image sample set may also be normalized to ensure that the sizes of the sample images input into the paired image generation model are the same.
[0119] In step S33, the unpaired image sample set is input into the first deep learning model for training to obtain a paired image generation model.
[0120] In step S34, the image of the person to be paired is input into the paired image generation model, and the target person image is output.
[0121] In step S35, according to the gender of the person in the target person image, irregular image feature adjustment is performed to conform to the gender characteristics of the person.
[0122] In step S36, the paired image sample set is input into the second deep learning model for training to obtain a character image gender conversion model.
[0123] The solution of this embodiment performs data cleaning on the unpaired image sample set to filter out abnormal character images in the unpaired image sample set; performs size normalization processing on the character images in the unpaired image sample set to make the sizes of the character images the same, and cleans the acquired unpaired image sample set, thereby improving the subsequent training accuracy of the paired image generation model and the training accuracy of the character image gender conversion model, providing a basis for the subsequent generation of rich magic expressions.
[0124] Figure 4 is a flowchart of an image generating method according to an exemplary embodiment. Figure 1 As shown, the image generation method can be executed by an image generation device, which can be implemented by software and / or hardware and used in an electronic device, which can be a computer, a server or a smart phone, etc. The method includes the following steps.
[0125] In step S41, at least one original character image is acquired.
[0126] In an optional implementation of this embodiment, at least one original character image provided by the user can be obtained from the client; wherein the client can be a smart phone, a tablet computer or a computer, etc., which is not limited in this embodiment. It should be noted that the magic expression involved in this embodiment refers to converting the user's facial state into other states (magic expressions), for example, converting the user's facial state into a cartoon image, an anime image or other 3D virtual image, etc., which is not limited in this embodiment.
[0127] In an optional implementation of this embodiment, when a user records a short video or takes a photo on a client, an image of a person in the short video recorded by the user or the photo taken by the user can be obtained, and the image of the person is the original image of the person.
[0128] It should be noted that when a user records a short video, since each short video includes multiple continuous video frames, each video frame can be regarded as an image. Therefore, multiple original character images can be obtained in the short video recorded by the user.
[0129] Exemplarily, in this embodiment, when a video or photo triggering instruction from the user is received, multiple original character images contained in the video data and the original character images contained in the taken photos can be acquired in real time.
[0130] In step S42, the original person image is input into the person image gender conversion model, and a target person image having a pairing relationship with the original person image is output.
[0131] Among them, the character image gender conversion model is trained according to the training method of the character image gender conversion model involved in any embodiment of the present disclosure; the pairing relationship represents: the correspondence between another character image and any character image obtained after the gender of any character image is changed and the background image is kept unchanged.
[0132] In an optional implementation of the present embodiment, after obtaining at least one original character image provided by the user through the client, the obtained original character image can be further input into the character image gender conversion model to generate a target character image that has a pairing relationship with the input original character image.
[0133] For example, if the original character image is an image of a male character with short hair, the image is input into the character image gender conversion model, and an image of a female character with long hair paired with the male character image with short hair can be output.
[0134] It should be noted that, in this embodiment, the original person image and the paired target person image have the same backgrounds and the same user IDs (Identity Documents) corresponding to the two images, except for the different genders of the two persons. It is understandable that if the two person images have the same backgrounds and the same user IDs corresponding to the two images, except for the opposite genders, then it can be determined that the two person images are paired images and have a paired relationship.
[0135] In an optional implementation of this embodiment, after obtaining the target person image paired with the original person image, the obtained target person image can be further displayed to the user through the client.
[0136] The solution of this embodiment obtains at least one original character image provided by the user through the client; inputs the original character image into the character image gender conversion model, and outputs a target character image that has a matching relationship with the original character image; solves the problem of poor flexibility in gender conversion of character images in related technologies, and can generate magic expressions opposite to the user's gender, and can generate magic expressions with rich images.
[0137] Figure 51 is a flow chart of a method for generating a magic expression character image according to an exemplary embodiment. This embodiment is a further refinement of the above technical solution. The technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 5 As shown, the method for generating a magic expression character image includes the following steps.
[0138] In step S51, at least one key frame is extracted from the continuous video frames provided by the user through the client according to a set rule, and the key frame is used as the original character image.
[0139] In an optional implementation of this embodiment, at least one key frame can be extracted from the continuous video frames provided by the user through the client according to the set rules, and the key frame is used as the original character image. The set rule can be a random extraction rule, for example, randomly extracting multiple video frames from the continuous video frames provided by the client, and using the extracted multiple video frames as key frames; the set rule can also be an interval extraction rule, for example, one video frame can be extracted every certain number (for example, 2 or 3, etc.) of video frames as a key frame; the set rule can also be a full extraction rule, for example, all video frames provided by the client are used as key frames. In this embodiment, key frames can also be extracted by other rules, which will not be repeated in this embodiment, and it is not a limitation of this embodiment.
[0140] It can be understood that each video frame can be regarded as an original character image, so in this embodiment, the key frame collected by any of the above rules can be used as the original character image.
[0141] In step S52, the original person image is input into the person image gender conversion model, and a target person image having a pairing relationship with the original person image is output.
[0142] In step S53, the facial key points and facial contours in the target person image are identified.
[0143] In an optional implementation of this embodiment, after obtaining the target person image paired with the original person image, the facial key points and facial contour in the target person image can be further identified; wherein the facial key points may include: eyes, nose, mouth, forehead or ears, etc., which are not limited in this embodiment.
[0144] Optionally, in this embodiment, the facial key points in the target person image can be identified by a facial key point detection model, and the facial contours in the target person image can be segmented by a semantic segmentation model.
[0145] Among them, the facial key point detection model can be a DCNN (Deep Convolutional Network), MTCNN (Multi-task Cascaded Convolutional Networks) or DAN (Deep Alignment Network) face key point detection model to detect the facial key points of the target person image. The semantic segmentation model can be a FCNs (Fully Convolution Networks), Mask R-CNN (Mask Region Convolutional Network) or PSPNet (Pyramid Scene Parsing Network) segmentation model to segment the face contour of the target person image.
[0146] In an optional implementation of this embodiment, relatively sparse facial key points, such as 68, 81 or 106, can be detected to locate relatively obvious and important feature points on the face (such as eye key points, eyebrow key points, nose key points, mouth key points, contour key points, etc.) to reduce the processing volume and processing time.
[0147] In step S54, the target person image is beautified according to the facial key points and facial contours to obtain a magic expression person image, which is updated as the target person image.
[0148] In an optional implementation of the present embodiment, after acquiring the facial key points and facial contour in the target person image, the target person image can be further beautified according to the facial key points and facial contour in the target person image to obtain a magic expression person image, and the person image after the beauty processing is determined to be updated as the target person image.
[0149] In an optional implementation of the present embodiment, beautifying the target person image according to facial key points and facial contours may include: identifying the beauty material selected by the user; determining the location to add the beauty material according to the facial key points and facial contours, and adding the beauty material to the location.
[0150] Among them, the beauty makeup materials can be eye shadow, lipstick, earrings, hair accessories, necklaces or pearl tears, etc., which are not limited in this embodiment.
[0151] In a specific implementation, after the client recognizes the beauty makeup material selected by the user, it can further determine the adding position of the beauty makeup material according to the acquired facial key points and facial contours, and add the beauty makeup material to the position according to the determined adding position.
[0152] For example, if the beauty makeup material selected by the user is eye shadow, the position of the eyelids can be determined based on the acquired facial key points and facial contours, and the eye shadow selected by the user can be added to the eyelids; if the beauty makeup material selected by the user is earrings, the position of the ears can be determined based on the acquired facial key points and facial contours, and the earrings selected by the user can be added to the ears.
[0153] In an optional implementation of the present embodiment, after the beauty materials selected by the user are added to the corresponding positions, it can be further confirmed whether the user is satisfied with the beauty processing. If not satisfied, the user can change the beauty materials, or the location where the beauty materials are added, by nodding or shaking his head, etc., which is not restricted in the present embodiment.
[0154] In an optional implementation of this embodiment, after obtaining the target person image after beauty processing, the person image of the key frames and connected frames in the continuous video frames collected on the client can be updated according to the target person image and displayed to the client.
[0155] The solution of this embodiment can realize beauty processing of the target person's image by identifying the facial key points and facial contour in the target person's image; determining the location to add the beauty material according to the facial key points and facial contour, and adding the beauty material to the location, thereby constructing a variety of different magic expressions, enriching the magic expressions, and improving the user experience.
[0156] In order to better understand the embodiments of the present disclosure, Figure 6 is a flowchart of a method for training a gender conversion model for a person image according to an exemplary embodiment. The method can be run in a server. Figure 6 , which mainly includes the following steps.
[0157] In step S61, a male person image dataset and a female person image dataset are collected.
[0158] Specifically, a large number of real images of men and women can be collected; further, the collected data sets are processed, for example, abnormal data are removed, the collected data sets are normalized, etc.
[0159] In step S62, the Cycle GAN network is trained.
[0160] The processed male and female image datasets are input into the Cycle GAN network for training.
[0161] In step S63, male and female paired image data are obtained.
[0162] After training, the Cycle GAN network can transform the gender of any user image, and the user ID information and background in the image before and after the transformation remain unchanged. Based on the trained Cycle GAN, a large number of image pairs before and after gender change, that is, male and female paired image data, can be generated.
[0163] In step S64, the male and female paired image data are optimized.
[0164] After observation, the male and female paired image data obtained in step S63 have the following problems:
[0165] 1. When changing gender, irregular hair will be generated. For example, when changing from a man to a woman, the video will show long hair that keeps flashing.
[0166] To solve this problem, this embodiment uses multiple 3D hairstyles for men and women designed by designers, and uses a camera posture estimation algorithm based on PNP to attach the 3D hairstyle to each image. At this time, the image data of men and women with fixed hairstyles are obtained.
[0167] 2. There are cases where some users' hair is exposed outside the fixed hairstyle, which seriously affects the overall sense of reality.
[0168] To solve this problem, this embodiment divides the exposed hair of the user into two parts: the exposed hair covering the face area, and the exposed hair covering the non-face area; for the exposed hair covering the face area, this embodiment designs a bangs elimination network, the core of which is to reconstruct the user's facial skin. Specifically, the skin of some areas of the user's face (such as the forehead, etc.) is masked (such as painted white), and the user's original image is used for supervision, so that the network can reconstruct the skin of the masked area. The specific process of eliminating the exposed hair covering the face area can be to cover the exposed hair area covering the face area of the user, and after the exposed hair covering the face area is calculated by the network, the masked area is reconstructed as the user's skin, and the exposed hair covering the face area is eliminated. For the exposed hair covering the non-face area, this embodiment applies the Inpainting related algorithm to eliminate the exposed hair covering the non-face area marked by the mask, and restore the corresponding semantic information such as the background.
[0169] In step S65, the Pix2Pix network is trained.
[0170] The optimized male and female paired image data is input into the Pix2Pix network to train the Pix2Pix network. After training, the network can convert the gender of any input user image while retaining the user's ID information and background information, and generate a fixed hairstyle on the user's head and remove the user's excess hair.
[0171] In order to better understand the embodiments of the present disclosure, Figure 7a is a flow chart of a method for generating a magic expression character image according to an exemplary embodiment. The method can be executed in real time on a mobile terminal. Figure 7a , which mainly includes the following steps.
[0172] In step S71, a user character image is obtained.
[0173] During actual operation, real-time frame extraction is performed for the short videos shot by users.
[0174] In step S72, a target person image paired with the user person image is acquired.
[0175] The user character image is input into Pix2Pix to obtain the target character image paired with the user character image.
[0176] In step S73, the key point network and the semantic segmentation network are called to obtain segmentation information such as face key points and face masks, and then combined with the corresponding materials to obtain the final result after gender transformation.
[0177] In order to better understand the embodiments of the present disclosure, Figure 7b-7d is a schematic diagram of a magic expression generated by a method for generating a magic expression character image according to an exemplary embodiment, from Figure 7b It can be seen from the figure that in this embodiment, the hairstyle of the character can be changed according to the needs of the user, and it is determined whether the user is satisfied with the replaced hairstyle. If the user is not satisfied, the hairstyle of the user can be changed by shaking his head or the like; Figure 7c It can be seen that in this embodiment, the makeup of the character can be changed according to the needs of the user, and it can be determined whether the user is satisfied with the makeup. If the user is not satisfied, the user's makeup can be changed by shaking his head. It should be noted that in this embodiment, the accessories or decorations worn by the user can also be changed. This will not be repeated in this embodiment. It can be understood that it is not a limitation of this embodiment. Figure 7d It can be seen that other magic expressions can also be generated in this embodiment, such as the girl next door, the yuppie, the domineering transformation, the male god, the goddess, the ever-changing male god, the ever-changing goddess, the cheongsam beauty and the school uniform boy, etc., which are not limited in this embodiment.
[0178] Figure 8 is a block diagram of a training device for a character image gender conversion model according to an exemplary embodiment. Figure 8 The device includes an acquisition module 81, a training module 82, an output module 83 and a training module 84.
[0179] An acquisition module 81 is configured to acquire an unpaired image sample set, wherein the unpaired image sample set includes a male character image set and a female character image set that have no pairing relationship;
[0180] The training module 82 is configured to input the unpaired image sample set into the first deep learning model for training to obtain a paired image generation model, wherein the paired image generation model is a recurrent adversarial generation network, and the paired image generation model is used to generate a person image with a paired relationship for each person image in the unpaired image sample set;
[0181] The output module 83 is configured to input the image of the person to be paired into the paired image generation model and output the target person image; wherein the image of the person to be paired and the target person image have a paired relationship and constitute a paired image sample;
[0182] The training module 84 is configured to input the paired image sample set into the second deep learning model for training to obtain a character gender conversion model, wherein the second deep learning model is a paired adversarial generative network, and the character gender conversion model is used to perform character gender conversion operations.
[0183] The solution of this embodiment is to obtain an unpaired image sample set through an acquisition module, wherein the unpaired image sample set includes a male character image set and a female character image set that have no pairing relationship; input the unpaired image sample set into the first deep learning model for training through the training module to obtain a paired image generation model; input the image of the character to be paired into the paired image generation model through the output module to output the target character image; input the paired image sample set into the second deep learning model for training through the training module to obtain a character gender conversion model, which solves the problem of poor flexibility in gender conversion of character images in related technologies, and can obtain paired image data with the opposite gender to the character image, providing a basis for subsequent generation of rich magic expressions.
[0184] Optionally, the training device for the character image gender conversion model also includes: an adjustment module, configured to adjust irregular image features that conform to the character's gender characteristics based on the character's gender in the target character image; wherein the irregular image features include at least one of the following: the character's hair, the clothes worn by the character, or the accessories worn by the character in the target character image.
[0185] Optionally, the adjustment module is specifically configured to identify the gender, head outline and posture of the person in the target person image;
[0186] According to the character's gender, head outline and posture, the replacement hairstyle is updated to the target character image to perform irregular image feature adjustment.
[0187] Optionally, the training device for the character image gender conversion model involved in this embodiment also includes: the device also includes: a recognition module, configured to identify the exposed hair of the character in the target character image, wherein the exposed hair of the character is the hair of the character outside the replacement hairstyle; when the exposed hair covers the face area, determining the skin color of the area covered by the exposed hair based on the skin color of the face not covered by the exposed hair to eliminate the exposed hair; and / or, when the exposed hair does not cover the face area, eliminating the exposed hair based on the semantic information of the image background not covered by the exposed hair.
[0188] Optionally, the person image to be paired involved in this embodiment is any image in the unpaired image sample set.
[0189] Optionally, the training device for the character image gender conversion model also includes: a cleaning module, configured to perform data cleaning on the unpaired image sample set to filter out abnormal character images in the unpaired image sample set, wherein the abnormal character images include at least one of the following images: images that do not contain facial features, or overexposed images; and / or, normalizing the size of each character image in the unpaired image sample set to make the size of each character image the same.
[0190] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0191] Fig. 9 is a block diagram of a device for generating a magic expression character image according to an exemplary embodiment. Fig. 9 The device includes an acquisition module 91 and an output module 92.
[0192] An acquisition module 91 is configured to acquire at least one original character image;
[0193] The output module 92 is configured to input the original character image into the character image gender conversion model, and output a target character image that has a pairing relationship with the original character image, wherein the character image gender conversion model is trained according to the training method of the character image gender conversion model involved in any embodiment of the present disclosure; the pairing relationship represents: the correspondence between any character image and another character image obtained by changing the gender of any character image while keeping the background image unchanged.
[0194] The solution of this embodiment obtains at least one original character image provided by the user through the client through the acquisition module; the original character image is input into the character image gender conversion model through the output module, and a target character image that has a matching relationship with the original character image is output, which solves the problem of poor flexibility in gender conversion of character images in related technologies, and can generate magic expressions opposite to the user's gender, and can generate magic expressions with rich images.
[0195] Optionally, the acquisition module 91 is specifically configured to extract at least one key frame from the continuous video frames provided by the user through the client according to a set rule, and use the key frame as the original character image;
[0196] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0197] Fig.10 FIG. 1 is a structural block diagram of an electronic device according to an exemplary embodiment. Fig.10 As shown, the electronic device includes a processor 101; a memory 102 for storing executable instructions of the processor 101, and the memory 102 may include a random access memory (RAM) and a read-only memory (ROM); wherein the processor 101 is configured to execute instructions to implement the above-mentioned training method of the character image gender conversion model, or the method for generating magic expression character images.
[0198] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 102 storing executable instructions. The above instructions can be executed by a processor 101 of an electronic device (server or smart terminal) to complete the above-mentioned training method of the character image gender conversion model, or the method of generating magic expression character images.
[0199] Alternatively, the storage medium may be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
[0200] In an exemplary embodiment, a computer program product is also provided. When the instructions in the computer program product are executed by a processor of an electronic device (server or smart terminal), the training method of the above-mentioned character image gender conversion model, or the method of generating magic expression character images is implemented.
[0201] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0202] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A training method for a character image gender conversion model, characterized in that: include: Acquire an unpaired image sample set, wherein the unpaired image sample set includes a male character image set and a female character image set that have no pairing relationship; The pairing relationship indicates: the correspondence between another character image obtained by changing the gender of any character image and keeping the background image unchanged and any of the character images; Inputting the unpaired image sample set into a first deep learning model for training to obtain a paired image generation model, wherein the first deep learning model is a recurrent adversarial generation network, and the paired image generation model is used to generate a person image having the paired relationship for each person image in the unpaired image sample set; Inputting the image of the person to be paired into the paired image generation model, and outputting the target person image; wherein the image of the person to be paired and the target person image have a paired relationship, and constitute a paired image sample; The paired image sample set is input into the second deep learning model for training to obtain a character gender conversion model, wherein the second deep learning model is a paired adversarial generation network, and the character gender conversion model is used to perform a character gender conversion operation; wherein the paired adversarial generation network is a Pix2Pix network.
2. The method according to claim 1, characterized in that Before the step of inputting the paired image sample set into the person image gender conversion model for training, the method further includes: According to the gender of the person in the target person image, irregular image feature adjustment is performed to conform to the gender characteristics of the person; wherein the irregular image feature includes at least one of the following: the hair of the person in the target person image, the clothes worn by the person, or the accessories worn by the person.
3. The method according to claim 2, characterized in that The step of adjusting the irregular image features in accordance with the gender of the person in the target person image comprises: Identify the gender, head outline and posture of the target person in the image; According to the gender of the character, the head outline of the character and the posture of the character, the replacement hairstyle is updated to the target character image to perform irregular image feature adjustment.
4. The method according to claim 3, characterized in that After the step of updating the replacement hairstyle to the target person image to perform the irregular image feature adjustment step, the method includes: Identifying exposed hair of a person in the target person image, wherein the exposed hair of the person is the hair of the person outside the replacement hairstyle; When the exposed hair covers the human face area, determining the skin color of the area covered by the exposed hair according to the skin color of the human face not covered by the exposed hair, so as to eliminate the exposed hair; and / or, When the exposed hair does not cover the human face area, the exposed hair is eliminated according to the image background not covered by the exposed hair.
5. The method according to claim 1, characterized in that Before inputting the unpaired image sample set into the paired image generation model for training, the method further includes: Performing data cleaning on the unpaired image sample set to filter out abnormal person images in the unpaired image sample set, wherein the abnormal person images include at least one of the following images: an image that does not contain facial features, or an overexposed image; and / or, The size of each person image in the unpaired image sample set is normalized to make the sizes of each person image the same.
6. An image generation method, characterized in that: include: Obtain at least one original character image; The original character image is input into a character image gender conversion model, and a target character image having a pairing relationship with the original character image is output, wherein the character image gender conversion model is trained according to the training method of the character image gender conversion model according to any one of claims 1 to 5; the pairing relationship represents: the correspondence between another character image obtained by changing the gender of any character image while keeping the background image unchanged and any of the character images.
7. A training device for a character image gender conversion model, characterized in that: include: An acquisition module is configured to acquire an unpaired image sample set, wherein the unpaired image sample set includes a male character image set and a female character image set that have no pairing relationship; The pairing relationship indicates: the correspondence between another character image obtained by changing the gender of any character image and keeping the background image unchanged and any of the character images; A training module, inputting the unpaired image sample set into a first deep learning model for training to obtain a paired image generation model, wherein the paired image generation model is a recurrent adversarial generation network, and the paired image generation model is used to generate a person image having the paired relationship for each person image in the unpaired image sample set; An output module is configured to input the image of the person to be paired into the paired image generation model and output a target person image; wherein the image of the person to be paired and the target person image have a paired relationship and constitute a paired image sample; The training module is configured to input the paired image sample set into the second deep learning model for training to obtain a character gender conversion model, wherein the second deep learning model is a paired adversarial generation network, and the character gender conversion model is used to perform a character gender conversion operation; wherein the paired adversarial generation network is a Pix2Pix network.
8. The device according to claim 7, characterized in that The device also includes: The adjustment module is configured to adjust the irregular image features in accordance with the gender of the person in the target person image; wherein the irregular image features include at least one of the following: the hair of the person in the target person image, the clothes worn by the person, or the accessories worn by the person.
9. The device according to claim 8, characterized in that The adjustment module is specifically configured to identify the gender, head outline and posture of the person in the target person image; According to the gender of the character, the head outline of the character and the posture of the character, the replacement hairstyle is updated to the target character image to perform irregular image feature adjustment.
10. The device according to claim 9, characterized in that The device also includes: A recognition module, configured to recognize exposed hair of a person in the target person image, wherein the exposed hair of the person is the hair of the person outside the replacement hairstyle; When the exposed hair covers the human face area, determining the skin color of the area covered by the exposed hair according to the skin color of the human face not covered by the exposed hair, so as to eliminate the exposed hair; and / or, When the exposed hair does not cover the human face area, the exposed hair is eliminated according to semantic information of the image background not covered by the exposed hair.
11. The device according to claim 7, characterized in that The device also includes: A cleaning module is configured to perform data cleaning on the unpaired image sample set to filter out abnormal person images in the unpaired image sample set, wherein the abnormal person images include at least one of the following images: an image that does not contain facial features, or an overexposed image; and / or, The size of each person image in the unpaired image sample set is normalized to make the sizes of each person image the same.
12. An image generating device, characterized in that: include: An acquisition module, configured to acquire at least one original character image; The output module is configured to input the original character image into the character image gender conversion model, and output a target character image having a pairing relationship with the original character image, wherein the character image gender conversion model is trained according to the training method of the character image gender conversion model described in any one of claims 1 to 5; the pairing relationship represents: the correspondence between another character image obtained by changing the gender of any character image while keeping the background image unchanged and any of the character images.
13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the command to implement the training method of the character image gender conversion model as described in any one of claims 1 to 5, or the image generation method as described in claim 6.
14. A storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the training method of the character image gender conversion model as described in any one of claims 1 to 5, or the image generation method as described in claim 6.
15. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the training method for a character image gender conversion model as described in any one of claims 1 to 5, or the image generation method as described in claim 6 is implemented.
