Method and apparatus for generating new portrait images from original images

Abstract

The present invention relates to the field of artificial intelligence. Specifically, it relates to a method for generating a new image of a person from an original image, the method comprising at least the steps of: providing a first original image comprising a person; identifying at least one pose key point of the person in the first original image; cropping a plurality of foreground image patches and a plurality of background image patches from the first original image based on the pose key point; extracting a first feature vector from the foreground image patches and the background image patches; obtaining a second feature vector comprising pose information of the person; and synthesizing a new image of the person from the first feature vector and the second feature vector. The method further comprises: inputting the first original image and a second original image comprising the person to a trained neural network model to synthesize the new image of the person, the new image of the person having the appearance of the person and the background in the first original image and the pose of the person in the second original image. It also relates to an apparatus for generating a new image of a person from an original image.

Description

TECHNICAL FIELD

[0001] The present application relates to a method for generating a new person image from an original image. The present application also relates to an apparatus for generating a new person image from an original image. BACKGROUND

[0002] In recent years, machine learning and deep learning have gradually become widely used algorithms in the fields of computer vision and autonomous driving. The performance of a machine learning model not only depends on the quality of model construction, but also requires the quality and quantity of training data. Research and analysis of training data and data augmentation are important research topics in recent years. In existing data augmentation methods, generative networks can generate new image data based on a training data set, and are the only method in data augmentation that can generate other new data in addition to the training set.

[0003] Although existing generative networks such as variational autoencoder (VAE) and generative adversarial networks (GAN) can generate new samples based on a training data set with a limited number of training samples, the quality of the generated new samples is often insufficient for use as training samples for machine learning and deep learning, and the generation process is mostly random, with little or no controllability of the target image style.

[0004] When generating person images using existing generative networks, the generated images have problems such as being unrealistic and having a disconnected foreground and background, and thus cannot be used as training samples for machine learning and deep learning. The reason for this is that existing generative networks do not have a controllable model that can control multiple parameters such as person pose, foreground (i.e., appearance), and background.

[0005] In addition, in the field of audio-visual entertainment, such as in the film and video production and electronic game production industries, there is a demand for "pose transfer" of a person, i.e., to transform the pose of a person in image A to the pose of a person in image B without changing the foreground and background of the person in image A. However, existing generative networks are difficult to achieve realistic and natural "pose transfer".

[0006] Therefore, it is desirable to provide a person image generation method that can achieve controllability of pose, foreground, and background in a person image and generate a person image in which the pose, foreground, and background are well integrated. SUMMARY

[0007] The object of the present application is achieved by providing a method for generating a new person image from an original image, the method comprising at least the following steps:

[0008] i) providing a first original image comprising a person;

[0009] ii) identifying at least one pose key point of the person in the first original image;

[0010] iii) cropping a plurality of foreground image patches and a plurality of background image patches from the first original image based on the at least one pose key point;

[0011] iv) extracting at least one first feature vector from the plurality of foreground image patches and the plurality of background image patches;

[0012] v) obtaining at least one second feature vector comprising person pose information; and

[0013] vi) synthesizing a new person image from the at least one first feature vector extracted in step iv) and the at least one second feature vector obtained in step v).

[0014] According to an optional embodiment of the present application, the plurality of foreground image patches collectively cover the complete human body and each foreground image patch respectively comprises a different human body part.

[0015] According to an optional embodiment of the present application, the plurality of background image patches are cropped from the first original image in the following way: a polygonal contour comprising the complete person is determined from the at least one pose key point, and then the area outside the polygonal contour in the first original image is divided into a plurality of background image patches.

[0016] According to an optional embodiment of the present application, step iv) is performed in the following way:

[0017] - normalizing all of the plurality of foreground image patches and the plurality of background image patches into the same size; and

[0018] - stitching all of the normalized foreground image patches and background image patches and inputting them into a feature extraction network to extract the at least one first feature vector.

[0019] According to an optional embodiment of the present application, the following step is performed after step ii) and before step iii): a cropping bounding box that highlights a single complete person is determined from the at least one pose key point, and then the first original image is cropped according to the cropping bounding box, wherein the cropped image is used as the basis for cropping in step iii).

[0020] According to an optional embodiment of the present application, step v) is performed in the following way:

[0021] a) connecting the identified pose key points according to a real human skeleton linking manner and then converting them into a visual person pose image; and

[0022] b) Extract the at least one second feature vector from the human pose image using an additional feature extraction network.

[0023] According to an optional embodiment of the present invention, the pose key points in step a) are determined from the person in the first original image in step ii).

[0024] According to an optional embodiment of the present invention, the pose key points in step a) are determined from the person in a second original image that is different from the first original image.

[0025] The present invention also achieves its objective through a method for generating new human figures from original images, the method comprising: inputting a first original image and a second original image containing a human figure into a trained neural network model to synthesize a new human figure image, the new human figure image having the background and human appearance from the first original image and the human pose from the second original image; the neural network model is trained in the following manner:

[0026] i') Provide training images containing people;

[0027] ii') Identify at least one pose keypoint of a person in the training image;

[0028] iii') Extract multiple foreground image blocks and multiple background image blocks from the training image based on the at least one pose key point;

[0029] iv') Extract at least one first feature vector from the plurality of foreground image blocks and the plurality of background image blocks;

[0030] v') Generate a visualized human pose image from the determined at least one pose key point and extract at least one second feature vector from the human pose image;

[0031] vi') A new person image is synthesized from the at least one first feature vector extracted in step iv') and the at least one second feature vector extracted in step v').

[0032] vii') Calculate the loss function between the new image of the person and the training image;

[0033] viii') Determine whether the training of the neural network model has reached the preset number of training steps. If the preset number of training steps has not been reached, backpropagate the loss function update gradient to the network parameters of the neural network model and continue training the neural network model; if the preset number of training steps has been reached, end the training of the neural network model.

[0034] The present invention also aims to provide an apparatus for generating a new image of a person from an original image, the apparatus comprising a processor and a computer-readable storage device communicatively connected to the processor, the computer-readable storage device storing a computer program which, when executed by the processor, is used to implement the method described above.

[0035] This invention enables the generation of high-quality new portrait images with foreground and background features of a first target object and the pose of a second target object, without paired annotations. The process involves decoupling and fusing the portrait pose with the foreground and background through steps such as pose extraction, region of interest (ROI) cropping, normalization, feature extraction, and fusion. These steps allow for controllability of the pose, foreground, and background during the portrait image generation process, and better integration of the foreground and background, solving the problems of uncontrollable generation processes, unrealistic image content, and foreground / background separation in existing methods.

[0036] Other advantages and advantageous embodiments of the subject matter of the invention will be apparent from the description, drawings and claims. Attached Figure Description

[0037] Further features and advantages of the present invention can be further illustrated by the following detailed description of specific embodiments with reference to the accompanying drawings. The drawings are as follows:

[0038] Figure 1 A schematic structural block diagram of an apparatus for generating a person image according to an exemplary embodiment of the present invention is shown.

[0039] Figure 2 A schematic structural block diagram of an apparatus for generating a person image according to an exemplary embodiment of the present invention is shown.

[0040] Figure 3 A flowchart illustrating a method for generating a portrait image according to an exemplary embodiment of the present invention is shown;

[0041] Figure 4 A flowchart illustrating one step of a method for generating a portrait image according to an exemplary embodiment of the present invention;

[0042] Figure 5 A flowchart illustrating another step of a method for generating a portrait image according to an exemplary embodiment of the present invention;

[0043] Figure 6 A flowchart illustrating another step of a method for generating a person image according to an exemplary embodiment of the present invention;

[0044] Figure 7 A flowchart illustrating another step of a method for generating a person image according to an exemplary embodiment of the present invention;

[0045] Figure 8 A flowchart illustrating a method for training a neural network model for generating new human images from an original image, according to an exemplary embodiment of the present invention;

[0046] Figure 9 The original image is shown according to an exemplary embodiment of the present invention;

[0047] Figure 10 The original image shown is illustrated with identified pose key points, a figure border, and a cropping border, according to an exemplary embodiment of the present invention.

[0048] Figure 11 The image shows a human figure in a pose according to an exemplary embodiment of the present invention;

[0049] Figure 12 This illustrates a foreground image block according to an exemplary embodiment of the present invention;

[0050] Figure 13 A background image block is shown according to an exemplary embodiment of the present invention;

[0051] Figure 14 A schematic diagram of the structure of a foreground-background feature extraction model (VAE) according to an exemplary embodiment of the present invention is shown; and

[0052] Figure 15 A schematic diagram of the structure of U-Net, a pose feature extraction model according to an exemplary embodiment of the present invention, is shown. Detailed Implementation

[0053] To make the technical problems to be solved, the technical solutions, and the beneficial technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and several exemplary embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of this invention. In the drawings, the same or similar reference numerals refer to the same or equivalent parts.

[0054] Figure 1 This diagram illustrates a schematic structural block diagram of an apparatus 100 for generating a new person image from an original image, according to an exemplary embodiment of the present invention. The apparatus 100 includes a processor 10 and a computer-readable storage device 20 communicatively connected to the processor 10. The computer-readable storage device 20 stores a computer program, which, when executed by the processor 10, implements a method for generating a person image, which will be explained in detail below.

[0055] According to an exemplary embodiment, a display device 30 is provided that is communicatively connected to the processor 10. With the aid of the display device 30, a user can view the original image processed by the device 100 and the new person image generated by the device 100.

[0056] According to an exemplary embodiment, an input device 40 is provided that is communicatively connected to the processor 10. With the aid of the input device 40, a user can select or input the original image to be processed by the device 100. The input device 40 may include, for example, a keyboard, a mouse, and / or a touchscreen.

[0057] According to an exemplary embodiment, a camera device 50 communicatively connected to the processor 10 is provided. With the aid of the camera device 50, a user can capture images of people as the original images to be processed by the device 100.

[0058] According to an exemplary embodiment, an original image set consisting of multiple images of people is provided. The original image set may be stored in a computer-readable storage device 20 or in a separate storage device communicatively connected to the processor 10.

[0059] Figure 2 A schematic structural block diagram of an apparatus 100' for generating a new person image from an original image is shown according to an exemplary embodiment of the present invention. Figure 2 The above is shown and combined with the following text. Figure 2 The features and details of the described device 100' can be understood as being Figure 1 As shown and in combination with the above Figure 1 The features and details of the described device 100 may be supplemented or substituted. In particular, these features and details may be combined in any suitable manner.

[0060] The apparatus 100' includes: a first processing unit 110 configured to acquire human pose key points 70 of the original image (see [reference]). Figures 10-11 ) and convert it into a visual image of a person in pose 90 (see Figure 11 The second processing unit 120 is configured to extract and normalize the foreground image block (also called the foreground ROI) and the background image block (also called the background ROI) of the original image; and the third processing unit 130 is configured to extract the foreground-background features of the first original image and the pose features of the person in the first or second original image and then generate a new person image from the extracted foreground-background features and pose features.

[0061] Figure 3 A flowchart is shown for a method 200 for generating a new person image from an original image according to an exemplary embodiment of the present invention.

[0062] In step S210, a first original image 80 containing the person 60 is provided, for example, as shown in the image below. Figure 9 As shown. For example, the first original image 80 can be any image from the set of original images mentioned above. Alternatively, the first original image can be a portrait image taken by the user using the camera device 50 or a frame of a portrait image extracted from a video stream.

[0063] Next, in step S220, at least one pose key point 70 of the person 60 in the first original image is identified using a pose key point recognition model. Figures 10-11 In the example shown, 18 pose keypoints 70 were identified, corresponding to the left and right eyes, left and right ears, mouth, neck joints, left and right shoulder joints, left and right elbow joints, left and right wrist joints, left and right hip joints, left and right knee joints, and left and right ankle joints. However, the pose keypoint recognition model can also be set to recognize more or fewer pose keypoints.

[0064] According to an exemplary embodiment, the pose keypoint recognition model is a pre-trained model obtained by training the Open Pose model using the MS COCO dataset. Alternatively, the pose keypoint recognition model can also be any other suitable model known in the art.

[0065] Next, in step S230, multiple foreground image blocks and multiple background image blocks are extracted from the first original image 80 based on the pose key points 70.

[0066] According to an exemplary embodiment, step S230 further includes (see...) Figure 4 ):

[0067] In step S231, multiple foreground image blocks are cropped from the first original image 80 based on the pose key points 70, so that the multiple foreground image blocks can collectively cover the entire human body, and each foreground image block includes different parts of the human body. In particular, each foreground image block contains as little or no human pose information as possible. By using this cropping method, the various parts of the human appearance can be decoupled, ensuring that the human appearance information is extracted independently of the human pose information.

[0068] In one exemplary embodiment, such as Figure 12 As shown, the multiple foreground image blocks include ten image blocks: head block 406, torso block 401 from left and right shoulder joints to left and right hip joints, left upper arm block 402, left forearm block 403, right upper arm block 407, right forearm block 408, left thigh block 404, left calf block 405, right thigh block 409, and right calf block 410.

[0069] Alternatively, the present invention may employ other suitable cropping schemes to crop foreground image blocks from pose keypoints 70. For example, more or fewer than ten foreground image blocks may be divided.

[0070] Additionally or alternatively, the captured foreground image blocks of people should contain as little background information as possible.

[0071] In step S232, multiple background image blocks are extracted from the first original image 80 based on the pose key points 70, such that the multiple background image blocks completely or almost completely overlap each other and cover the entire background area of ​​the original image. This method decouples the various parts of the image background, ensuring that each background image block contains as little or no information related to the person as possible, thereby ensuring the extraction of background information independent of the person's pose and position in the image.

[0072] According to an exemplary embodiment, the contour of the entire human body, such as a polygonal contour, especially a polygonal contour with right angles, can be determined from the pose key points 70, and then the area outside the contour in the first original image 80 can be divided into multiple background image blocks, such as multiple background image blocks arranged successively along the contour line of the human body.

[0073] In one exemplary embodiment, such as Figure 13 As shown, the multiple background image blocks include eight image blocks: the upper left block 412, the upper middle block 411, the upper right block 415, the middle left block 413, the middle right block 416, the lower left block 414, the lower middle block 418, and the lower right block 417.

[0074] Alternatively, the present invention may employ other suitable methods to extract background image blocks from pose keypoints 70. For example, more or fewer than eight background image blocks may be divided.

[0075] Additionally or alternatively, the captured background image block of the person should contain as little foreground information as possible or none at all.

[0076] It should be noted that steps S231 and S232 can be executed in any order or simultaneously.

[0077] Optionally, after performing step S220 to identify the human pose key points 70 in the first original image 80 and before performing step S230 to extract multiple foreground image blocks and multiple background image blocks from the first original image 80, the following cropping step S221 can be performed on the first original image 80: the first original image 80 is cropped according to the human pose key points 70 determined in step S220, so that the cropped image prominently contains a single complete person. The cropped image is used as the basis for extracting the foreground and background image blocks in the subsequent step S230.

[0078] In one exemplary embodiment, such as Figure 10 As shown, the human figure outline 51 is determined by the human pose key points 70 identified in step S220 (see...). Figure 10 The inner dashed border in the image), and the character border 51 is expanded outward to form a cropping border 52 for cropping the first original image 80 (see the ...). Figure 10 (The outer dashed border in the image). For example, the outward expansion ratio can be 1.5 times (that is, the side length of the cropping border 52 is 1.5 times that of the character border 51) or any other suitable multiple.

[0079] Next, in step S240, at least one first feature vector is extracted from the plurality of foreground image blocks 401-410 and the plurality of background image blocks 411-418. Accordingly, the extracted first feature vector contains foreground information (i.e., person's appearance information) and / or background information from the first original image 80. Since the previous segmentation of the plurality of foreground image blocks and the plurality of background image blocks has decoupled the person's foreground-background information from the person's pose information, the first feature vector extracted here does not contain the person's pose information.

[0080] In an exemplary embodiment, step S240 further includes (see...) Figure 5 ):

[0081] In step S241, all the foreground image blocks 401-410 and the background image blocks 411-418 are normalized to the same size;

[0082] Next, in step S242, all normalized image blocks 401-418 are input into the foreground-background feature extraction model to extract the at least one first feature vector.

[0083] In one example, all normalized image blocks 401-418 can be stitched together in a preset order and then input into the foreground-background feature extraction model. The stitching method includes, but is not limited to, stacking like a pyramid, and / or stitching together into a single image.

[0084] In one exemplary embodiment, any suitable form of neural network, such as a VAE, can be used to construct the foreground-background feature extraction model. Exemplarily, the extracted foreground-background feature vector is a feature vector sampled from the intermediate layers of the VAE, such as... Figure 14 As shown.

[0085] Then, in step S250, at least one second feature vector containing the person's pose information is obtained. This step can be performed in two ways.

[0086] In the first execution method, the at least one second feature vector is obtained by determining the at least one second feature vector from the human posture key points 70 in the first original image 80 identified in step S220.

[0087] In the second execution method, the at least one second feature direction is obtained in the following manner (see...). Figure 6 ):

[0088] In step S251, a second original image containing a person is provided, which is different from the first original image 80. For example, the second original image may be derived from the original image set mentioned above. Alternatively, the second original image may be a person image captured by the user using the camera device 50 or a frame of a person image extracted from a video stream.

[0089] In step S252, the pose key points of the person in the second original image are determined. The content explained above for step S220 also applies to step S252.

[0090] In step S253, the at least one second feature vector is determined from the pose key points of the person in the second original image.

[0091] For both of the above methods, after determining the human pose key points from the original image, the at least one second feature vector can be determined from the human pose key points using the following method (see...). Figure 7 ):

[0092] In step S2501, the obtained pose key point data is connected according to the real-world human skeleton connection method and converted into a visualized human pose image 90, such as an RGB format human pose image. Figure 11 As shown;

[0093] Then, in step S2502, the person pose image 90 obtained in step S2501 is used, for example, to extract the at least one second feature vector by inputting it into a pose feature extraction model. Exemplarily, any suitable form of neural network can be used, such as U-Net (e.g., ...). Figure 15 (As shown) to construct a pose feature extraction model.

[0094] In one example, each body skeleton part in the human pose image 90 is represented by a corresponding color, for example, a different color from each other.

[0095] The steps S2501 and S2502 explained above are applicable to extracting a second feature vector from any human image, such as the first original image 80 and the second original image mentioned above, based on pose key points.

[0096] Next, in step S260, a new image of a person is synthesized from the at least one first feature vector extracted in step S240 and the at least one second feature vector obtained in step S250.

[0097] When the second feature vector is extracted from the first original image 80, the first feature vector used to synthesize the new person image, i.e., the foreground-background feature vector and the second feature vector, i.e., the pose feature vector, all originate from the same first original image 80. Therefore, the new person image generated in this way is essentially a reconstruction of the first original image 80.

[0098] When the second feature vector is extracted from the second original image, the first feature vector used to synthesize the new person image, i.e., the foreground-background feature vector, comes from the first original image, while the second feature vector, i.e., the pose feature vector, comes from the second original image, which is different from the first original image 80. Therefore, the resulting new person image is essentially equivalent to transferring the pose of the person in the second original image to the first original image 80. The person in the new image has the clothing, appearance, and background of the person in the first original image 80, but has the pose of the person in the second original image.

[0099] In an exemplary embodiment, step S260 can be performed by inputting the at least one first feature vector and the at least one second feature vector into a preset generative network to synthesize a new person image. The preset generative network may be, for example, U-Net or other suitable neural networks. Exemplarily, the U-Net here is skipped to the corresponding layer of the U-Net in step S2502.

[0100] Figure 8 A flowchart is shown of a method 200' for training a neural network model according to an exemplary embodiment of the present invention, the neural network model being constructed for generating new human images from original images. Method 200' has the same characteristics as described above. Figure 3 The explained method 200 has the same steps S210, S220, S221, S230, S240 and S260. Therefore, these steps can be referred to the above description and will not be repeated here.

[0101] Step S250' is performed using the first execution method of step S250 as described above, specifically: determining at least one second feature vector from the pose key points of the person in the training image 80 identified in step S220. The features and details explained above regarding the first execution method of step S250 are applicable to step S250' and will not be repeated here. In particular, step S250' may include the features described above. Figure 7 The steps S2501 and S2502 are explained.

[0102] After the new image of the person, i.e., the restored image of the training image, is generated in step S260, in step S270', the feature difference between the new image of the person and the training image, i.e., the loss function, is calculated. The feature difference calculated in this step is used to update the parameters of the neural network.

[0103] According to an exemplary embodiment, the loss function includes the following three parts:

[0104] The first part is the content loss (feature L2 loss) of a pre-trained network on MS COCO based on VGG19.

[0105]

[0106] Where F represents the network feature parameters, and X represents the original input image. For the generated restored image, {l c} represents the feature layer corresponding to VGG19;

[0107] The second part is the KL divergence distance.

[0108]

[0109] Where X′ is the foreground ROI obtained in step S231, X″ is the background ROI obtained in step S232, Y is the input human pose image, and z is the extracted overall image feature. and These represent the unsampled (mean) feature vectors obtained from the intermediate layers of the feature extraction network in steps S240 and S250, respectively.

[0110] The third part is the style loss (Gram matrix L2 loss) based on the VGG19 pre-trained network.

[0111]

[0112] Where G is the Gram matrix of network feature parameters, {l s} represents the feature layer corresponding to VGG19.

[0113] These three parts are combined with appropriate weights to obtain the loss function for the generative network. The goal of training the generative network is to minimize this loss function. This loss function design aims to preserve image content and style while mitigating the influence of local image information. It also decouples pose features from foreground and background features, allowing for separate control of these parameters during image generation.

[0114] The present invention can also employ other appropriate methods to design the loss function.

[0115] Then, in step S280', it is determined whether the training of the neural network model used to generate the person image has reached a preset number of training steps. If the preset number of training steps has not been reached, the loss function update gradient is backpropagated to all network parameters of the neural network model used to generate the person image, and the training of the neural network model continues. Otherwise, if the preset number of training steps has been reached, the training of the neural network model used to generate the person image ends.

[0116] The trained neural network model for generating human images can be used to perform the above-mentioned references. Figures 3-15 The methods explained for generating images of people include incremental methods for training data and methods for human pose transfer in the field of audiovisual entertainment.

[0117] While some embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the invention. The appended claims and their equivalents are intended to cover all modifications, substitutions, and alterations that fall within the scope and spirit of the invention.

Claims

1. A method (200) for generating a new image of a person from an original image, the method comprising at least the following steps: i) Provide a first original image (80) containing the person; ii) Identify at least one pose key point (70) of the person in the first original image (80); iii) Based on the at least one pose key point (70), extract multiple foreground image blocks and multiple background image blocks from the first original image (80); iv) Extract at least one first feature vector from the plurality of foreground image blocks and the plurality of background image blocks that does not contain human pose information; v) Obtain at least one second feature vector containing human posture information; and vi) A new image of a person is synthesized from the at least one first feature vector extracted in step iv) and the at least one second feature vector obtained in step v).

2. The method (200) according to claim 1, characterized in that, The multiple foreground image blocks together cover the complete human body, and each foreground image block includes different parts of the human body.

3. The method (200) according to claim 1 or 2, characterized in that, The multiple background image blocks are extracted from the first original image (80) in the following manner: a polygonal outline containing a complete human figure is determined by the at least one pose key point (70), and then the area in the first original image (80) located outside the polygonal outline is divided into multiple background image blocks.

4. The method (200) according to claim 1 or 2, characterized in that, Perform step iv) in the following manner: - Normalize all the foreground image blocks and the background image blocks to the same size; and - Consolidate all normalized foreground image blocks and all background image blocks and input them into a feature extraction network to extract at least one first feature vector.

5. The method (200) according to claim 3, characterized in that, Perform step iv) in the following manner: - Normalize all the foreground image blocks and the background image blocks to the same size; and - Consolidate all normalized foreground image blocks and all background image blocks and input them into a feature extraction network to extract at least one first feature vector.

6. The method (200) according to any one of claims 1-2 and 5, characterized in that, After step ii) and before step iii), the following steps are performed: determining a cropping border (52) that prominently contains a single complete figure by the at least one pose key point (70) and then cropping the first original image (80) based on the cropping border (52), wherein the cropped image is used as the basis for the cropping in step iii).

7. The method (200) according to claim 3, characterized in that, After step ii) and before step iii), the following steps are performed: determining a cropping border (52) that prominently contains a single complete figure by the at least one pose key point (70) and then cropping the first original image (80) based on the cropping border (52), wherein the cropped image is used as the basis for the cropping in step iii).

8. The method (200) according to claim 4, characterized in that, After step ii) and before step iii), the following steps are performed: determining a cropping border (52) that prominently contains a single complete figure by the at least one pose key point (70) and then cropping the first original image (80) based on the cropping border (52), wherein the cropped image is used as the basis for the cropping in step iii).

9. The method (200) according to any one of claims 1-2, 5, 7-8, characterized in that, Perform step v) in the following manner: a) Connect the identified pose key points according to the linking method of the real human skeleton and then convert them into a visualized human pose image (90); and b) The at least one second feature vector is extracted from the human pose image (90) using an additional feature extraction network.

10. The method (200) according to claim 3, characterized in that, Perform step v) in the following manner: a) Connect the identified pose key points according to the linking method of the real human skeleton and then convert them into a visualized human pose image (90); and b) The at least one second feature vector is extracted from the human pose image (90) using an additional feature extraction network.

11. The method (200) according to claim 4, characterized in that, Perform step v) in the following manner: a) Connect the identified pose key points according to the linking method of the real human skeleton and then convert them into a visualized human pose image (90); and b) The at least one second feature vector is extracted from the human pose image (90) using an additional feature extraction network.

12. The method (200) according to claim 6, characterized in that, Perform step v) in the following manner: a) Connect the identified pose key points according to the linking method of the real human skeleton and then convert them into a visualized human pose image (90); and b) The at least one second feature vector is extracted from the human pose image (90) using an additional feature extraction network.

13. The method (200) according to claim 9, characterized in that, The pose key points in step a) are determined from the person in the first original image (80) in step ii).

14. The method (200) according to any one of claims 10-12, characterized in that, The pose key points in step a) are determined from the person in the first original image (80) in step ii).

15. The method (200) according to claim 9, characterized in that, The pose key points in step a) are determined from the figure in a second original image that is different from the first original image (80).

16. The method (200) according to any one of claims 10-12, characterized in that, The pose key points in step a) are determined from the figure in a second original image that is different from the first original image (80).

17. A method (200) for generating a new image of a person from an original image, the method comprising: A first original image and a second original image containing a person are input into a trained neural network model to synthesize a new person image. The new person image incorporates the background and appearance of the person from the first original image and the pose of the person from the second original image. The neural network model is trained in the following manner: i') Provide training images containing people; ii') Identify at least one pose key point of a person in the training image (70); iii') Based on the at least one pose key point (70), extract multiple foreground image blocks and multiple background image blocks from the training image; iv') Extract at least one first feature vector that does not contain human pose information from the plurality of foreground image blocks and the plurality of background image blocks; v') Generates a visualized human pose image (90) from the determined at least one pose key point (70) and extracts at least one second feature vector from the human pose image (90); vi') A new person image is synthesized from the at least one first feature vector extracted in step iv') and the at least one second feature vector extracted in step v'). vii') Calculate the loss function between the new image of the person and the training image; viii') Determine whether the training of the neural network model has reached the preset number of training steps. If the preset number of training steps has not been reached, backpropagate the loss function update gradient to the network parameters of the neural network model and continue training the neural network model; if the preset number of training steps has been reached, end the training of the neural network model.

18. An apparatus (100) for generating a new image of a person from an original image, the apparatus (100) comprising a processor (10) and a computer-readable storage device (20) communicatively connected to the processor (10), the computer-readable storage device (20) storing a computer program which, when executed by the processor (10), is used to implement the method (200) according to any one of the preceding claims.

Images