Method, system and program carrier for processing images of persons

Abstract

The present application relates to the field of computer vision and the field of autonomous driving. A method for processing a person image, the method comprising the steps of: obtaining an original person image; extracting first feature information and second feature information from the original person image, the second feature information being different from the first feature information; and training a person image generation model based on the first feature information and the second feature information, in combination with using a variational autoencoder and a generative adversarial network. The present application also provides a method for generating a person image by means of a person image generation model, a system for processing a person image and a machine-readable program carrier. The present application aims to model the influence between different feature information of a person image in combination with considering GAN and VAE, and to optimize the training process of a person image generation model by means of automatic decoupling of different features.

Description

Technical Field

[0001] This invention relates to a method for processing human images, a method for generating human images using a human image generation model, a system for processing human images, and a machine-readable program carrier. Background Technology

[0002] In human figure recognition, the performance of a neural network directly depends on the network structure itself and the richness of training samples used during training. More realistic and natural image training samples can significantly improve the generalization ability and recognition accuracy of the neural network. However, obtaining realistic training samples requires extensive collection, filtering, and annotation work, and the resulting datasets are often imbalanced, which greatly increases the difficulty of training the neural network. To better train neural networks, existing technologies have proposed using specific human figure image generation models to generate human figures as training samples for the neural network.

[0003] Existing technologies typically employ generative networks such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) to generate more images. However, existing generative networks are usually trained using only one method. For example, using only VAEs often leads to blurry and unrealistic images; using only GANs often results in distorted image content and uncontrollable generation processes. Therefore, for tasks such as object detection, pose recognition, and person identification, which not only demand high overall performance from neural networks but also require a large quantity, high quality, and diverse range of images, existing generative networks cannot meet these requirements. Summary of the Invention

[0004] The purpose of this invention is to provide a method for processing human images, a method for generating human images using a human image generation model, a system for processing human images, and a machine-readable program carrier.

[0005] According to a first aspect of the present invention, a method for processing portrait images is provided, the method comprising the following steps:

[0006] S1: Obtain the original image of the person;

[0007] S2: Extract first feature information and second feature information from the original image of the person, wherein the second feature information is different from the first feature information; and

[0008] S3: Based on the first and second feature information, train the character image generation model by combining variational autoencoder and generative adversarial network.

[0009] This invention specifically includes the following technical concept: by combining VAE and GAN during training, not only can the accurate restoration of high-frequency information in human images be ensured through generative adversarial processes, but it also overcomes the problems of image content blurring, lack of detail, and limited generation variety that occur when VAE and GAN are used separately. Furthermore, by extracting the first and second features of the human figure through a feature extraction network, and then fusing these features in the generator, automatic decoupling and fusion between different human figure features are achieved, ultimately providing a controllable human image generation process.

[0010] Optionally, step S3 includes:

[0011] S31: Based on the first feature information and the second feature information, a restored image of a person is generated by using a generative network of a generative adversarial network, wherein a combination of a variational autoencoder and a U-Net network is used as the generative network.

[0012] S32: Input the restored image and the original image into the discriminative network of the generative adversarial network to determine whether they are real or fake;

[0013] S33: Adjust at least one parameter of the portrait image generation model based on the deviation between the restored portrait image and the original portrait image.

[0014] Optionally, step S2 includes: extracting human pose feature vectors and human appearance feature vectors from the original human image using a variational autoencoder network and / or a U-Net network.

[0015] Here, since only the pose and appearance information of the person are used during training, the additional person masking technique to subtract the foreground and background of the person is eliminated, thus saving costs. Furthermore, based on the optimization of the training process for person image generation using a generative adversarial network architecture, this invention proposes to introduce a variational autoencoder in feature extraction. This eliminates the need for decoupled learning of the target person under different poses or appearances during training, enabling end-to-end training directly on the image dataset.

[0016] Optionally, before performing step S2, the method further includes the following steps:

[0017] Segmenting the human figure region from the original human figure image;

[0018] Extract key point information of the person in the person area; and

[0019] The key information of the figure is stitched together and visualized to obtain the figure's posture image.

[0020] Optionally, segmenting the person region from the original person image includes: cropping an image block larger than the person's bounding box from the original person image to ensure that part of the background of the person image appears in the cropped image block.

[0021] Optionally, step S33 includes: calculating a loss function based on the restored image of the person and the discrimination result of the discriminative network; and adjusting at least one parameter of the image generation model according to the loss function.

[0022] Optionally, the loss function used for generative networks includes:

[0023] The first part is the content loss (feature L2 loss) of the pre-trained network on MSCOCO based on VGG19, and its formula is expressed as follows:

[0024]

[0025] Where F represents the VGG19 network feature parameters, and X represents the original input image of the person. To generate a restored image of the person, {l c} represents the feature layer corresponding to VGG19.

[0026] The second part is the KL divergence distance, which is expressed by the following formula:

[0027]

[0028] Where X is the original input image of the person, Y is the input image of the person's pose, and z is the extracted overall image features. and These represent the unsampled (mean) feature vectors obtained in the intermediate layers of the feature extraction network used.

[0029] The third part is the style loss (Gram matrix L2 loss) based on the VGG19 pre-trained network, and its formula is expressed as follows:

[0030]

[0031] Where F represents the VGG19 network feature parameters, G represents the Gram matrix of the VGG19 network feature parameters, and X represents the original input image of the person. To generate a restored image of the person, {l S} represents the feature layer corresponding to VGG19.

[0032] The fourth part is the FM (Feature Matching) loss of the discriminative network, and its formula is expressed as follows:

[0033]

[0034] Where D represents the network feature parameters of the discriminative network, and X represents the original input image of the person. To generate a restored image of the person, {l D} represents the feature layer corresponding to the discriminative network. The calculation of this FM loss is based on the output of the discriminative network. FM (feature matching) can be understood as requiring that the features extracted from the image generated by the generative network when it passes through the discriminative network be as close as possible to (match) the features extracted from the original image of the person when it passes through the discriminative network.

[0035] Alternatively, the loss function used for the discriminative network can be expressed by the following equation:

[0036]

[0037] Where X is the original image of the person being input. The generated restored image of the person is given by Y, which is the input image of the person's pose, and E. X，Y and Let Dis(X,Y) and Dis(X,Y) represent the expected value and Dis(X,Y) respectively. These represent the output results of the discriminative network.

[0038] According to a second aspect of the present invention, a method for generating a person image using a person image generation model is provided, wherein the person image generation model is the person image generation model described in the method according to the first aspect, the method comprising the following steps:

[0039] Obtain feature information from the first person's image;

[0040] Obtain another feature information from the second person's image; and

[0041] The feature information and the other feature information are input into the person image generation model to generate a target person image, wherein the target person image has the feature information of the first person image and the other feature information of the second person image.

[0042] According to a third aspect of the present invention, a system for processing portrait images is provided, the system being used to perform the method according to a first aspect of the present invention, the system comprising:

[0043] An image acquisition module, configured to acquire original human images;

[0044] A feature extraction module is configured to extract first feature information and second feature information from an original image of a person, wherein the second feature information is different from the first feature information; and

[0045] The training module is configured to train the person image generation model based on the first feature information and the second feature information, by combining variational autoencoder and generative adversarial network.

[0046] According to a fourth aspect of the invention, a machine-readable program carrier is provided having a computer program stored thereon, the computer program being configured to implement the method according to the first aspect of the invention when run on a computer. Attached Figure Description

[0047] The invention will now be described in more detail with reference to the accompanying drawings, which will provide a better understanding of its principles, features, and advantages. The drawings include:

[0048] Figure 1 A flowchart illustrating a method for processing portrait images according to an exemplary embodiment of the present invention is shown;

[0049] Figure 2 A flowchart illustrating the feature extraction steps of a method for processing portrait images according to an exemplary embodiment of the present invention is shown.

[0050] Figure 3 The key points and pose of a person in an image extracted according to the method of the present invention are shown.

[0051] Figure 4 A schematic diagram of the feature extractor (VAE) used in the method of the present invention is shown;

[0052] Figure 5 A schematic diagram of another feature extractor, U-Net, used in the method of the present invention is shown;

[0053] Figure 6 A flowchart illustrating a method for generating a person image using a person image generation model according to an exemplary embodiment of the present invention is shown; and

[0054] Figure 7 A block diagram of a system for processing human images according to an exemplary embodiment of the present invention is shown. Detailed Implementation

[0055] To make the technical problems to be solved, the technical solutions, and the beneficial technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and several exemplary embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of this invention.

[0056] Figure 1 A flowchart of a method for processing human images according to an exemplary embodiment of the present invention is shown.

[0057] In step S1, the original image of the person is obtained.

[0058] In the context of this invention, the original human image is particularly understood as an image of a person selected (manually or mechanically) from real images, such as those acquired by an image acquisition device.

[0059] In the field of autonomous driving, raw images of people can be captured and stored in a storage medium using image acquisition devices (e.g., cameras) mounted on the vehicle. Alternatively, the raw images may be captured by one or more cameras in the vehicle's surrounding environment, or the raw image may be a composite image of images captured from multiple cameras at different angles. Optionally, the acquisition or storage process may have fixed time intervals, and the number of frames captured for the same scene should not differ significantly from the number of frames captured for other scenes. Optionally, during the acquisition process, images can be distinguished, in particular, by using uniquely coded timestamps to prevent similar individuals from appearing multiple times or repeatedly.

[0060] In step S2, first feature information and second feature information are extracted from the original image of the person, wherein the second feature information is different from the first feature information.

[0061] Here, the first feature information represents, for example, the person's pose information, and the second feature information represents, for example, the person's appearance information. For instance, the original person image can be processed and then input into a feature extraction network (e.g., an encoder network) to obtain the person's pose feature vector and appearance feature vector. There are many types of encoder networks; for example, a U-Net network can be used for extracting person's pose information, while a VAE network can be introduced for extracting person's appearance information to achieve automatic decoupling between different features of the person image.

[0062] In step S3, a person image generation model is trained based on the first feature information and the second feature information by combining variational autoencoder and generative adversarial network.

[0063] Specifically, for example, in step S31, a reconstructed image of a person can be generated using a generative network based on the first and second feature information and a generative adversarial model. In particular, the extracted feature vectors can be input into a pre-defined generative network to generate the target image (here referring to the reconstructed image of a person). For example, a combination of U-Net and VAE networks can be used as the generative network, and the U-Net network portion used in generating the reconstructed image can be skipped to the corresponding layers of the U-Net network portion used in the feature extraction step.

[0064] In step S32, the restored image and the original image can be input into a discriminative network of a generative adversarial network (GAN) to determine their authenticity. For example, a Markov discriminator (Patch-GAN) can be used. This discriminator consists entirely of convolutional layers, outputting an n×n matrix. The mean of the output matrix is ​​then used as the true / false output. Based on this discriminator, each image patch in the original image can be individually judged for authenticity and then weighted and averaged to arrive at a final overall judgment: whether the generated image and the original image are the same person.

[0065] In step S33, at least one parameter of the person image generation model is adjusted based on the deviation between the restored person image and the original person image. For example, the feature difference between the restored person image generated in step S32 and the original person image is calculated, and then the calculated feature difference is used to update the parameters of the person image generation model. Exemplarily, a loss function can be calculated based on the restored person image and the discrimination result of the discriminative network, and the generative adversarial network is iteratively trained continuously in response to the calculated loss function value until the loss function value converges or a preset number of training steps is reached. At this point, the finally trained generative network can be determined as the person image generation model.

[0066] Optionally, such as Figure 1 As exemplarily shown, in step S33, it can be further determined whether the training of the human image generation model has reached the preset number of training steps. If it has not reached the preset number of training steps, the loss function is updated and the gradient is backpropagated to all network parameters, and the features are extracted again to train the model.

[0067] There are many ways to design a loss function. In this embodiment of the invention, the loss function used for the generative network part includes four parts:

[0068] The first part is the content loss (feature L2 loss) of the pre-trained network on MSCOCO based on VGG19, and its formula is expressed as follows:

[0069]

[0070] Where F represents the VGG19 network feature parameters, and X represents the original input image of the person. To generate a restored image of the person, {l c} represents the feature layer corresponding to VGG19.

[0071] The second part is the KL divergence distance, which is expressed by the following formula:

[0072]

[0073] Where X is the original input image of the person, Y is the input image of the person's pose, and z is the extracted overall image features. and These represent the unsampled (mean) feature vectors obtained in the intermediate layers of the feature extraction network used.

[0074] The third part is the style loss (Gram matrix L2 loss) based on the VGG19 pre-trained network, and its formula is expressed as follows:

[0075]

[0076] Where F represents the VGG19 network feature parameters, G represents the Gram matrix of the VGG19 network feature parameters, and X represents the original input image of the person. To generate a restored image of the person, {l S} represents the feature layer corresponding to VGG19.

[0077] The fourth part is the FM (Feature Matching) loss of the discriminative network, and its formula is expressed as follows:

[0078]

[0079] Where D represents the network feature parameters of the discriminative network, and X represents the original input image of the person. To generate a restored image of the person, {l D} represents the feature layer corresponding to the discriminative network. The calculation of this FM loss is based on the output of the discriminative network. FM (feature matching) can be understood as requiring that the features extracted from the image generated by the generative network when it passes through the discriminative network be as close as possible to (match) the features extracted from the original image of the person when it passes through the discriminative network.

[0080] These terms are combined to form the loss function for the generative network, which aims to minimize this loss function. The purpose of this loss function design is to decouple pose and appearance features while preserving image content and style, allowing control over these parameters during image generation. The discriminative network's FM loss, on the other hand, greatly preserves the realism of high-frequency components, resulting in higher-quality generated images. It also stabilizes the training process of the discriminative network, leading to eventual convergence. Many similar loss function designs exist, and this invention does not limit the specific design or proportion of this particular loss function, provided it fulfills all the required functions.

[0081] In this exemplary embodiment, the loss function used for the discriminative network is log loss, and its formula is as follows:

[0082]

[0083] Where X is the original image of the person being input. The generated restored image of the person is given by Y, which is the input image of the person's pose, and E. X，Y and Let Dis(X,Y) and Dis(X,Y) represent the expected value and Dis(X,Y) respectively. These represent the output results of the discriminative network.

[0084] The purpose of the discriminative network is to maximize this loss function, thereby enabling it to distinguish between real and fake images. Furthermore, during training, it can engage in adversarial training with the generator, indirectly prompting the generator to achieve better generation results. Many similar loss function designs exist, and this invention does not limit the specific design and proportion of this loss function, provided that the discriminative network functionality is available.

[0085] Figure 2 A flowchart illustrating the feature extraction steps of a method for human image processing according to an exemplary embodiment of the present invention is shown.

[0086] In step S201, the person region is segmented from the acquired original person image. Here, for example, the image can be cropped into an image block that contains the complete person and is larger than the person's bounding box, to ensure that part of the background of the original person image appears in the cropped person image block.

[0087] In step S202, key point information of the person is extracted from the person region. Here, key point information that has been manually annotated can be extracted directly, or a pre-trained human pose estimation model can be used to detect the coordinates of key points of the person in the image. Specifically, there are many methods for detecting key points of a person's pose. This invention uses a pre-trained model trained on MSCOCO using the OpenPose detection method. However, the detection method can also be any method capable of detecting key points of a person's pose in an image; this embodiment of the invention does not impose specific limitations on the above detection method.

[0088] In step S203, the key point information of the figure is stitched together and visualized to obtain a figure posture image. Here, the obtained key point information of the figure posture can be stitched together according to the connection method of the human skeleton in real life, and visualized using RGB images to obtain the figure posture image. This invention does not impose specific limitations on the specific stitching order and visualization processing method.

[0089] In step S204, the obtained human pose image and the human image blocks segmented in step S201 are respectively input into the corresponding encoder network for feature vector extraction to obtain human appearance feature vector and human pose feature vector.

[0090] Figure 3 The diagram illustrates key points and poses of a person extracted from an image according to the method of the present invention. Figure 3 The left side shows key points of a person extracted using the method of this invention. Figure 3 The right side shows a human figure pose image formed by connecting key points of the figure according to the skeleton and then visualizing the image.

[0091] Figure 4 A schematic diagram of the feature extractor (VAE) used in the method of the present invention is shown.

[0092] The goal of the feature extraction step is to perform semantic segmentation on image patches, that is, to independently classify each pixel using the image patches surrounding it in order to interpret the required feature vector. In this embodiment, a VAE network is used to extract the appearance feature vector of a person from the original image. The VAE network includes an encoder part 402 and a decoder part 406. The encoder part 402 is used to encode the input image data 401 into a mean vector 403 and a variance vector 404, and the decoder part 406 is used to recover the image data 407 from these vectors. In addition, this VAE network allows for the addition of latent variables from the outside to intervene by sampling from the variance vector 405 and adding it to the mean vector, thereby achieving automatic decoupling of the appearance feature vector and the pose feature vector of the person. Compared with conventional autoencoders, the variational autoencoder used in this embodiment does not require additional decoupled learning of the target person. Instead, it can directly constrain the appearance feature vector extracted by the encoder and the introduced conditional feature vector (pose feature vector), so that the feature vector and the conditional feature vector are independent of each other in each dimension, so as to automatically achieve good separation of human posture and appearance.

[0093] In the method according to the present invention, the segmented human image block 401 is input into the VAE network, and the feature vector 405 after sampling by the intermediate layer of the VAE network is extracted as the desired human appearance feature vector.

[0094] Figure 5 A schematic diagram of another feature extractor, U-Net, used in the method of the present invention is shown.

[0095] In this embodiment, a portion of the U-Net network is used as a feature extraction network to extract the human pose feature vector, while another portion of the U-Net network is used to generate the restored image. For example... Figure 5 As shown, the U-Net network consists of two parts. The first part, the encoder 502, is used to gradually reduce the spatial dimension in order to extract features from the input image 501. The second part, the decoder 504, is used to upsample the image to gradually restore the details and spatial dimension of the object, thus forming the output image 505. A shortcut connection is typically present between the encoder 502 and the decoder 504, which helps the decoder better restore the details of the target.

[0096] In the method according to the present invention, the stitched human pose feature image 501 is input into the U-Net network, and after passing through the encoder, the corresponding human pose feature vector 503 is obtained. Then, the output of the VAE network (human appearance feature vector) is coupled with the intermediate layer of the U-Net network to ensure that the restored image generated by the conditional U-Net not only has the fixed representation (appearance features) of the original input image, but also incorporates the latent representation of random poses, so that the generated image shows consistency with the extracted pose features.

[0097] Figure 6 A flowchart illustrating a method for generating a person image using a person image generation model according to an exemplary embodiment of the present invention is shown. Here, the person image generation model is used in the method for person image processing according to the present invention.

[0098] In step S61, feature information of the first person image is obtained. For example, this feature information may be the person's pose feature vector. Here, for example, a pre-trained model can be used to extract keypoint information from the person image and visualize it as a person pose image. Then, for example, a corresponding feature extraction network can be used to extract the person pose feature vector from the person pose image.

[0099] In step S62, another feature information of the second person image is obtained. For example, this other feature information may be the person's appearance feature vector. Here, for instance, a pre-trained model can be used to extract key point information of the person, and the person region can be segmented from the person image. Then, the image patch containing the person region is input into a feature extraction network to separate the corresponding appearance feature vector.

[0100] In step S63, the aforementioned feature information and another feature information are input into the person image generation model to generate a target person image. Here, the feature information extracted in steps S61 and S62 is fused based on the trained person image generation model to obtain the target person image. For example, the target person image has the pose features of the first person image and the appearance features of the second person image.

[0101] Figure 7 A block diagram of a system for processing human images according to an exemplary embodiment of the present invention is shown.

[0102] like Figure 7 As shown, system 70 includes:

[0103] Image acquisition module 71 is configured to acquire original human images.

[0104] The feature extraction module 72 is configured to extract first feature information and second feature information from the original human image, wherein the second feature information is different from the first feature information.

[0105] Training module 73 is configured to train a person image generation model based on the first feature information and the second feature information, by combining variational autoencoder and generative adversarial network.

[0106] As an example, training module 73 may include restoration module 731, which is configured to generate restored human images using a generative network of generative adversarial network based on human pose information and human appearance information.

[0107] The training module 73 may also include a discrimination module 732, which is configured to input the restored image of the person and the original image of the person into the discriminative network of the generative adversarial network to distinguish between the real and the fake.

[0108] The training module 73 may also include a parameter adjustment module 733, which is configured to adjust at least one parameter of the person image generation model based on the deviation between the restored person image and the original person image.

[0109] Although specific embodiments of the invention have been described in detail herein, they are given for illustrative purposes only and should not be construed as limiting the scope of the invention. Various substitutions, alterations, and modifications can be conceived without departing from the spirit and scope of the invention.

Claims

1. A method for human image processing, the method comprising the steps of: S1: obtaining an original human image; S2: extracting first feature information and second feature information from the original human image, the second feature information being different from the first feature information, extracting the first feature information in the form of a human appearance feature vector from the original human image by means of a variational autoencoder network, obtaining a human pose image from the original human image and then extracting the second feature information in the form of a human pose feature vector from the human pose image by means of a U-Net network, constraining the human appearance feature vector extracted in the variational autoencoder and the human pose feature vector as a conditional feature vector such that the human appearance feature vector and the human pose feature vector are independent of each other in each dimension, so as to realize automatic decoupling of the human appearance feature vector and the human pose feature vector; and S3: training a human image generation model based on the first feature information and the second feature information in combination with using a variational autoencoder and a generative adversarial network.

2. The method of claim 1, wherein, The step S3 comprises: S31: generating a restored human image by means of a generative network of a generative adversarial network based on the first feature information and the second feature information, wherein a combination of a variational autoencoder and a U-Net network is used as the generative network; S32: inputting the restored human image and the original human image into a discriminative network of the generative adversarial network to discriminate authenticity; S33: adjusting at least one parameter of the human image generation model according to a deviation between the restored human image and the original human image.

3. The method according to claim 1 or 2, before performing the step S2, the method further comprises the steps of: segmenting a human region from the original human image; extracting human keypoint information in the human region; stitching and visualizing the human keypoint information to obtain a human pose image.

4. The method of claim 2, wherein, The step S33 comprises: calculating a loss function based on the restored human image and a discrimination result of the discriminative network; and adjusting at least one parameter of the human image generation model according to the loss function.

5. The method of claim 4, wherein, The loss function used for the generative network comprises: a first part represented by the following equation: Wherein, F is the VGG19 network feature parameter, X is the input original person image, is the generated restored person image, c is the VGG19 corresponding feature layer; a second part represented by the following equation: Wherein, X is the input original character image, Y is the input character posture image, z is the extracted image overall feature, and respectively represent the unsampled feature vectors obtained at the middle layer of the used feature extraction network; a third part represented by the following equation: Wherein, F is the VGG19 network feature parameter, G is the Gram matrix of the VGG19 network feature parameter, X is the input original person image, is the generated restored person image, s is the VGG19 corresponding feature layer; a fourth part represented by the following equation: Wherein, D is the network feature parameter of the discriminant network, X is the input original person image, is the generated restored person image, D is the feature layer corresponding to the discriminant network.

6. The method of claim 4, wherein, The loss function used for the discriminative network is represented by the following equation: wherein X is an input original person image, is a generated restored person image, Y is an input person pose image, E x,Y and respectively represent expectations, Dis(X, Y) and respectively represent output results of the discriminative network.

7. A method for generating an image of a person by means of a model for generating images of persons, wherein The human image generation model is the human image generation model in the method according to any one of claims 1 to 6, the method comprising the steps of: obtaining feature information of a first human image; obtaining another feature information of a second human image; and inputting the feature information and the other feature information into the human image generation model to generate a target human image, wherein the target human image has the feature information of the first human image and has the other feature information of the second human image.

8. A system for human image processing, the system being configured to perform the method according to any one of claims 1 to 6, the system comprising: an image acquisition module configured to be able to acquire an original person image; a feature extraction module configured to be able to extract first feature information and second feature information from the original person image, the second feature information being different from the first feature information; and a training module configured to be able to train the person image generation model based on the first feature information and the second feature information, in combination with using a variational autoencoder and a generative adversarial network.

9. A machine readable program carrier having stored thereon a computer program for implementing the method according to any one of claims 1 to 6 when it is run on a computer.

Images