Image generation method, device, electronic device and storage medium

Abstract

The present disclosure relates to an image generation method, device, electronic device and storage medium, and belongs to the field of image processing technology. The method first obtains features of multiple scales in a first image and a second image for indicating a body part of an object, and then deforms multiple first features of the first image based on optical flow information of multiple scales between the first image and the second image to achieve the purpose of changing the posture of the first object, and then generates an image based on the deformed features. Since the deformation in the above-mentioned image generation process is performed while retaining the complete feature information of the image, the accuracy of the texture in the generated target image is guaranteed when a large-scale posture change is performed, thereby improving the quality of the generated target image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image generating method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of image processing technology, many users hope to transform the images they take into different forms through image processing technology. For example, some users hope to transform the posture or clothes of the characters in the original image to generate a new image. The transformation of the character's clothes is the image texture transformation. In the related art, the image generation method generally adopts the attribute-decomposed gan (ADGAN) based on attribute decomposition. This method generates the target image after the posture and texture are changed by inputting the original image, the target posture and the target texture image into the ADGAN network.

[0003] Among the above technologies, the ADGAN method directly combines the target texture features with the target posture features. When a large posture change is performed, erroneous textures will appear in the generated target image, and the generated target image quality is poor. Summary of the invention

[0004] The present disclosure provides an image generation method, device, electronic device and storage medium, which can ensure the accuracy of texture in the generated target image and improve the quality of the generated target image when a large posture change is performed. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image generating method, the method comprising:

[0006] Acquire a plurality of first features of the first image, the plurality of first features comprising features corresponding to a plurality of body parts of a first object in the first image, wherein the features corresponding to each body part of the first object comprise features of a plurality of scales;

[0007] Acquire a second image, where the second image is used to provide a target posture of the first object;

[0008] Acquire optical flow information between the second image and the first image, where the optical flow information is used to represent deformation information of the same pixel in different images;

[0009] Based on the optical flow information, deforming the plurality of first features to obtain a plurality of second features;

[0010] The multiple second features are processed to obtain a target image.

[0011] The technical solution provided by the embodiment of the present disclosure first obtains features of multiple scales in the first image and the second image for indicating the body part of the object, and then deforms multiple first features of the first image based on the optical flow information of multiple scales between the first image and the second image to achieve the purpose of changing the posture of the first object, and then generates an image based on the deformed features. Since the deformation in the above-mentioned image generation process is performed while retaining the complete feature information of the image, the accuracy of the texture in the generated target image is guaranteed when a large posture change is performed, thereby improving the quality of the generated target image.

[0012] In some embodiments, the acquiring of the plurality of first features of the first image comprises:

[0013] Based on the first image, acquiring images of various body parts of the first object in the first image;

[0014] The plurality of first features are acquired based on images of various body parts of the first object.

[0015] Through the images of various parts of the body, the first features of different body parts are obtained respectively, so that when the posture is changed subsequently, it is only necessary to deform the features of the corresponding body parts respectively to achieve the posture change of the entire object, thereby improving the controllability of the features.

[0016] In some embodiments, the acquiring the plurality of first features based on the images of the respective body parts of the first object comprises:

[0017] The images of each body part of the first object are respectively input into the feature extraction sub-model, and for any body part image of the first object, first features of multiple scales output by multiple network layers in the feature extraction sub-model are obtained.

[0018] For each body part image, first features at multiple scales are obtained, which retains more information in the image, thereby making the quality of the target image generated based on the deformed first features higher.

[0019] In some embodiments, for any body part image of the first object, obtaining first features of multiple scales output by multiple network layers in the feature extraction submodel includes:

[0020] Acquire human semantic features of multiple scales of the body part, where the human semantic features of multiple scales are features extracted based on a general object dataset;

[0021] Inputting the body part image of the first object into the first-level network layer in the feature extraction sub-model, obtaining the first feature of the first scale output by the first-level network layer, and using the first feature of the first scale and the human body semantic feature of the first scale as input data of the next-level network layer;

[0022] For the first feature of any scale output by any level of network layer, the first feature of the scale is concatenated with the human body semantic feature of the same scale, and the concatenated first feature of the scale is input into the next level of network layer to obtain the first feature of the next level of scale;

[0023] In response to the scale of the first feature of the scale being the smallest scale except the smallest scale, the first feature of the scale is concatenated with a human body semantic feature of the same scale, and the concatenated first feature of the scale is input into the last network layer to obtain the first feature with the smallest scale.

[0024] By introducing the features of general object datasets, it can not only guide the extraction of human features and provide some low-level features, but also avoid the overfitting phenomenon caused by extracting features based on limited samples.

[0025] In some embodiments, based on the optical flow information, deforming the plurality of first features to obtain the plurality of second features comprises:

[0026] Based on the second image, obtaining masks of various body parts of the second object in the second image;

[0027] Based on the masks of the respective body parts of the second object and the masks of the respective body parts of the first object, acquiring optical flow information corresponding to the respective body parts;

[0028] Based on the optical flow information corresponding to each body part, the multiple first features are deformed to obtain the multiple second features.

[0029] By deforming the features through optical flow information, implicit deformation of the image is achieved. Different from explicit deformation that directly deforms the image, implicit deformation can alleviate the problem of difficulty in finding corresponding points during large posture changes, thereby improving the quality of the generated target image during large posture changes.

[0030] In some embodiments, the acquiring optical flow information of each body part based on the mask of each body part of the second object and the mask of each body part of the first object includes:

[0031] According to the multiple scales, respectively scale the masks of the respective body parts of the second object and the masks of the respective body parts of the first object to obtain the masks of the respective body parts of the second object at the multiple scales and the masks of the respective body parts of the first object at the multiple scales;

[0032] Based on the masks of the respective body parts of the second object at the same scale and the masks of the respective body parts of the first object, optical flow information of multiple scales corresponding to the respective body parts is obtained.

[0033] By acquiring optical flow information of multiple scales of each body part, the first features of multiple scales of each body part can be deformed simultaneously, so that the features of different scales can be effectively used for posture transformation in the subsequent image generation process.

[0034] In some embodiments, the processing of the plurality of second features to obtain a target image includes:

[0035] The second features of the same scale are fused to obtain fused features of multiple scales;

[0036] Perform texture normalization processing on the smallest scale fusion feature to obtain the smallest scale texture normalization feature, and perform convolution processing on the smallest scale texture normalization feature;

[0037] For the fused features of any scale except the smallest scale, the fused features of the scale are concatenated with the texture normalized features of the previous scale after convolution to obtain the concatenated features of the scale, and the texture normalization processing is performed on the concatenated features of the scale to obtain the texture normalized features of the scale;

[0038] In response to the texture normalization feature of the scale being the texture normalization feature of the largest scale, convolution processing is performed on the texture normalization feature of the largest scale to obtain a target image.

[0039] By processing the second features of multiple scales, a target image is obtained, so that the generated target image contains more information and has richer image details, thereby achieving the purpose of improving the quality of the target image.

[0040] In some embodiments, the method further comprises:

[0041] When texture normalization is performed on features corresponding to any scale, the normalization parameter used is determined based on an image obtained by scaling the image of the clothing portion in the first image, and the normalization parameter is used to represent the texture features of the clothing portion.

[0042] Performing a scale change on the image of the clothing portion in the first image to obtain the clothing portion images of the plurality of scales;

[0043] Extracting features of the clothing part images at multiple scales to obtain texture features at the multiple scales;

[0044] The texture normalization processing for features corresponding to any scale includes:

[0045] Based on the mean and variance of each element in the feature of the scale, the feature of the scale is normalized to obtain the normalized feature of the scale;

[0046] The texture features of the scale are fused with the normalized features of the scale to obtain the texture normalized features of the scale.

[0047] By performing texture normalization operation on the features, the texture in the first image is fitted into the target image, so that the generated target image can maintain texture details, thereby achieving the purpose of improving the quality of the generated target image.

[0048] In some embodiments, before deforming the plurality of first features based on the optical flow information to obtain the plurality of second features, the method further comprises:

[0049] Acquire a plurality of third features of a third image, the plurality of third features comprising features corresponding to a plurality of body parts of a third object in the third image, the features corresponding to each body part of the third object comprising features of the plurality of scales, and the third object in the third image comprising a target body part;

[0050] Based on the third feature corresponding to the target body part in the plurality of third features, the first feature corresponding to the target body part in the plurality of first features is replaced, and the target body part includes one or more body parts of the third object.

[0051] By replacing the first feature corresponding to the target body part with the third feature of the target body part in the third image, the feature of the body part to be generated can be replaced with the first feature, thereby achieving a change in the body part in the first object.

[0052] In some embodiments, the third image and the second image are the same image; or, the third image and the second image are different images.

[0053] The solution proposed in the present application can achieve changes in the body part or posture of the first object based on one image or two images.

[0054] In some embodiments, acquiring the second image includes:

[0055] Acquire at least one image from a reference video, and use the image as the second image;

[0056] After the plurality of second features are processed to obtain a target image, the method further comprises:

[0057] A target video is generated based on the target image corresponding to the at least one image.

[0058] The solution proposed in the present application can obtain at least one target image by referring to at least one image obtained from a video, based on changes in a body part or posture of a first object, and then generate a target video based on the at least one target image.

[0059] According to a second aspect of an embodiment of the present disclosure, there is provided an image generating device, the device comprising:

[0060] an acquisition unit configured to acquire a plurality of first features of the first image, the plurality of first features comprising features corresponding to a plurality of body parts of a first object in the first image, wherein the features corresponding to each body part of the first object comprise features of a plurality of scales;

[0061] The acquisition unit is configured to acquire a second image, where the second image is used to provide a target posture of the first object;

[0062] The acquisition unit is configured to acquire optical flow information between the second image and the first image, where the optical flow information is used to represent deformation information of the same pixel in different images;

[0063] A deformation unit is configured to perform deformation on the plurality of first features based on the optical flow information to obtain a plurality of second features;

[0064] The processing unit is configured to process the multiple second features to obtain a target image.

[0065] In some embodiments, the acquisition unit is configured to acquire, based on the first image, images of various body parts of the first object in the first image; and acquire the multiple first features based on the images of various body parts of the first object.

[0066] In some embodiments, the acquisition unit includes:

[0067] The input subunit is configured to input each body part image of the first object into the feature extraction submodel respectively, and for any body part image of the first object, obtain first features of multiple scales output by multiple network layers in the feature extraction submodel.

[0068] In some embodiments, the input subunit is configured to execute the acquisition of human semantic features of multiple scales of the body part, where the human semantic features of multiple scales are features extracted based on a general object data set; input the image of the body part of the first object into the first-level network layer in the feature extraction submodel to obtain the first feature of the first scale output by the first-level network layer, and use the first feature of the first scale and the human semantic features of the first scale as input data of the next-level network layer; for the first feature of any scale output by any-level network layer, splice the first feature of the scale with the human semantic features of the same scale, and input the spliced ​​first feature of the scale into the next-level network layer to obtain the first feature of the next-level scale; in response to the scale of the first feature of the scale being the smallest scale except the smallest scale, splice the first feature of the scale with the human semantic features of the same scale, and input the spliced ​​first feature of the scale into the last network layer to obtain the first feature of the smallest scale.

[0069] In some embodiments, the deformation unit comprises:

[0070] an acquisition subunit, configured to acquire masks of various body parts of the second object in the second image based on the second image;

[0071] The acquisition subunit is configured to execute, based on the masks of the respective body parts of the second object and the masks of the respective body parts of the first object, to acquire the optical flow information corresponding to the respective body parts;

[0072] The deformation subunit is configured to perform deformation on the multiple first features based on the optical flow information corresponding to each body part to obtain the multiple second features.

[0073] In some embodiments, the acquisition subunit is configured to perform scale changes on the masks of each body part of the second object and the masks of each body part of the first object according to the multiple scales, respectively, to obtain masks of each body part of the second object at the multiple scales and masks of each body part of the first object at the multiple scales; based on the masks of each body part of the second object and the masks of each body part of the first object at the same scale, obtain optical flow information of multiple scales corresponding to the each body part.

[0074] In some embodiments, the processing unit includes:

[0075] A fusion subunit is configured to fuse the second features of the same scale to obtain fused features of multiple scales;

[0076] The normalization subunit is configured to perform texture normalization processing on the fused feature with the smallest scale to obtain the texture normalized feature with the smallest scale, and perform convolution processing on the texture normalized feature with the smallest scale;

[0077] The splicing subunit is configured to execute, for the fusion features of any scale except the smallest scale, splicing the fusion features of the scale with the texture normalization features of the previous scale that have been processed by convolution to obtain the splicing features of the scale, and performing texture normalization processing on the splicing features of the scale to obtain the texture normalization features of the scale;

[0078] The processing subunit is configured to execute, in response to the texture normalization feature of the scale being the texture normalization feature of the largest scale, performing convolution processing on the texture normalization feature of the largest scale to obtain a target image.

[0079] In some embodiments, the apparatus further comprises:

[0080] a scale change unit configured to perform scale change on the image of the clothing part in the first image to obtain the clothing part images of the plurality of scales; and perform feature extraction on the clothing part images of the plurality of scales to obtain texture features of the plurality of scales;

[0081] The normalization subunit is configured to perform normalization processing on the features of the scale based on the mean and variance of each element in the features of the scale to obtain the normalized features of the scale; and to fuse the texture features of the scale with the normalized features of the scale to obtain the texture normalized features of the scale.

[0082] In some embodiments, the acquisition unit is configured to acquire a plurality of third features of a third image, the plurality of third features comprising features corresponding to a plurality of body parts of a third object in the third image, the features corresponding to each body part of the third object comprising features of the plurality of scales, and the third object in the third image comprising a target body part;

[0083] The device also includes:

[0084] A replacement unit is configured to replace a first feature corresponding to the target body part among the multiple first features based on a third feature corresponding to the target body part among the multiple third features, where the target body part includes one or more body parts of the third object.

[0085] In some embodiments, the third image and the second image are the same image; or, the third image and the second image are different images.

[0086] In some embodiments, the acquisition unit is configured to acquire at least one image from a reference video and use the image as the second image;

[0087] The device also includes:

[0088] The generating unit is configured to generate a target video based on the target image corresponding to the at least one image.

[0089] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0090] one or more processors;

[0091] A memory for storing program codes executable by the processor;

[0092] The processor is configured to execute the program code to implement the above-mentioned image generation method.

[0093] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, which includes: when the program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned image generation method.

[0094] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising a computer program, which implements the above-mentioned image generation method when executed by a processor.

[0095] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0097] Figure 1 is a schematic diagram of an implementation environment of an image generation method according to an exemplary embodiment;

[0098] Figure 2 is a flow chart of an image generating method according to an exemplary embodiment;

[0099] Figure 3 is a schematic diagram of an image generating method according to an exemplary embodiment;

[0100] Figure 4 is a model structure diagram of an image generation method according to an exemplary embodiment;

[0101] Figure 5 is a flow chart of an image generating method according to an exemplary embodiment;

[0102] Figure 6 is a schematic diagram showing a basic principle of feature extraction according to an exemplary embodiment;

[0103] Figure 7 is a schematic diagram showing a basic principle of optical flow deformation according to an exemplary embodiment;

[0104] Figure 8 is a schematic diagram of a basic principle of texture normalization according to an exemplary embodiment;

[0105] Fig. 9 is a flow chart of an image generating method according to an exemplary embodiment;

[0106] Fig.10 is a schematic diagram showing a basic principle of feature replacement according to an exemplary embodiment;

[0107] Fig.11 is a comparison diagram of the results of the present technical solution and the ADGAN method according to an exemplary embodiment;

[0108] Fig.12 is a result diagram of an image generation method according to an exemplary embodiment;

[0109] Fig.13 is a block diagram of an image generating device according to an exemplary embodiment;

[0110] Fig.14 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0111] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0112] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0113] The data involved in the present disclosure may be data authorized by the user or fully authorized by all parties.

[0114] Figure 1 is a schematic diagram of an implementation environment of an image generation method provided by an embodiment of the present disclosure, see Figure 1 , the implementation environment includes: a terminal 101 and a server 102.

[0115] The terminal 101 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop, a virtual reality terminal, an augmented reality terminal, a wireless terminal, and a laptop computer. The terminal 101 has a communication function and can access the Internet. The terminal 101 may generally refer to one of a plurality of terminals. This embodiment is only illustrated by the terminal 101. Those skilled in the art may know that the number of the above terminals may be more or less. The terminal 101 may run various applications for image processing functions, which may be video processing applications, social applications, or image processing applications.

[0116] The server 102 may be an independent physical server, or a server cluster or distributed file system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The server 102 may be associated with a database for storing image data and feature data, wherein the image data includes the first image, the second image, the third image, the generated target image, and other publicly disclosed image data sets uploaded by the user, and the feature data includes image semantic annotation information and human body semantic features obtained based on the publicly disclosed image data sets. The server 102 may be directly or indirectly connected to the terminal 101 via wired or wireless communication, which is not limited in the embodiments of the present application. Optionally, the number of the above-mentioned servers 102 may be more or less, which is not limited in the embodiments of the present application. Of course, the server 102 may also include other functional servers to provide more comprehensive and diversified services.

[0117] Figure 2 is a flowchart of an image generating method according to an exemplary embodiment. Figure 2 As shown, the method is executed by the server and includes the following steps:

[0118] In step 201, the server obtains a plurality of first features of a first image, the plurality of first features including features corresponding to a plurality of body parts of a first object in the first image, and the features corresponding to each body part of the first object include features of a plurality of scales.

[0119] The first object in the first image is a person, and the body parts of the first object are hair, face, top, arms, feet, etc.

[0120] In step 202, the server acquires a second image, where the second image is used to provide a target posture of the first object.

[0121] Among them, the second image includes a second object, the second object is a person, and the posture of the second object is the target posture. The server can transform the posture of the first object in the first image based on the posture of the second object in the second image, so that the posture of the first object can be consistent with the posture of the second object in the second image.

[0122] In step 203, the server obtains optical flow information between the second image and the first image, where the optical flow information is used to represent deformation information of the same pixel in different images.

[0123] The optical flow information can represent deformation information between the posture of the first object and the posture of the second object, including optical flow information between each body part of the first object and each body part of the second object, and the optical flow information of each body part includes optical flow information of multiple scales.

[0124] In step 204, the server deforms the multiple first features based on the optical flow information to obtain multiple second features.

[0125] In some embodiments, the server deforms the first features of multiple scales of each body part of the first object based on the optical flow information to obtain the multiple second features. By deforming the multiple first features, the posture of the first object is changed to the target posture.

[0126] In step 205, the server processes the multiple second features to obtain a target image.

[0127] The posture of the object in the target image is the target posture.

[0128] The technical solution provided by the embodiment of the present disclosure first obtains features of multiple scales in the first image and the second image for indicating the body part of the object, and then deforms multiple first features of the first image based on the optical flow information of multiple scales between the first image and the second image to achieve the purpose of changing the posture of the first object, and then generates an image based on the deformed features. Since the deformation in the above-mentioned image generation process is performed while retaining the complete feature information of the image, the accuracy of the texture in the generated target image is guaranteed when a large posture change is performed, thereby improving the quality of the generated target image.

[0129] Figure 3 is a schematic diagram of an image generating method according to an exemplary embodiment. Figure 3 As shown, this method enhances the texture details of the generated target image by extracting multi-scale features and texture normalization operations, and models the deformation information between the original image and the target image through optical flow information, thereby ensuring the accuracy of the texture in the generated target image.

[0130] Figure 4 is a model structure diagram of an image generation method according to an exemplary embodiment. The image generation model can change the body part and posture of the object in the input image. This embodiment is described by taking the posture change of the object in the input image as an example. Figure 4 As shown, the image generation model consists of the following parts:

[0131] (1) Semantic Segmentation Sub-Model

[0132] The semantic segmentation sub-model is used to perform semantic segmentation on the input image and obtain the semantic annotation information of the input image, such as Figure 4 As shown in part 401, the server can obtain images of various body parts of the object in the input image based on the semantic annotation information, and the images of various body parts are used for subsequent feature extraction steps. In some embodiments, the semantic segmentation submodel is a fully convolutional neural network (FCN), that is, the semantic segmentation submodel includes multiple convolutional layers, or a pyramid scene parsing network (PSPNet). Of course, the above-mentioned semantic segmentation submodel can also be other deep learning models, which is not limited in this embodiment.

[0133] (2) Feature extraction sub-model

[0134] The feature extraction submodel is used to extract features from the images of various body parts of the object in the input image. The feature extraction submodel includes multiple network layers. For any image of a body part, the feature extraction submodel obtains features of multiple scales of the image of the body part through multiple network layers, such as Figure 4 As shown in part 402, the features of the body part image at multiple scales are used in the subsequent step of deforming the features. In some embodiments, the feature extraction submodel is a learnable visual geometry group (VGG) or other deep learning model, which is not limited in this embodiment.

[0135] (3) Optical flow deformation sub-model

[0136] The optical flow deformation sub-model is used to obtain the optical flow information between the input image and the target pose image. The pose of the object in the target pose image is the target pose. The optical flow information is used to represent the deformation information of the same pixel in different images, such as Figure 4 As shown in part 403. In some embodiments, the server obtains multiple optical flow information by inputting masks of various body parts of the object in the input image and masks of various body parts of the object in the target posture image into the optical flow deformation sub-model, and deforms (Warp) the features of multiple scales of various body parts of the object in the input image based on the multiple optical flow information, so as to achieve the purpose of changing the posture of the object in the input image into the target posture. This part of the content is described in detail in the following further explanation of this solution, see Figure 5 and Fig. 9 The corresponding embodiment content.

[0137] (4) Texture Normalization Sub-Model

[0138] The texture normalization sub-model includes a first convolutional neural network and a second convolutional neural network. The texture normalization sub-model fuses the features of each body part of the object in the deformed input image of the same scale through the first convolutional neural network, performs texture normalization processing on the fused features of multiple scales through the second convolutional neural network, and performs convolution processing on the texture normalized features to obtain a target image, such as Figure 4 As shown in part 404, the posture of the object in the target image is the target posture. This part of the content is described in detail in the following further explanation of this solution in the embodiment of the present application, see Figure 5 and Fig. 9 The corresponding embodiment content.

[0139] The image generation method provided in this embodiment is based on Figure 4 The image generation model shown in the figure changes the posture of the object in the input image to obtain the target image. Figure 5This method is described. Figure 5 is a flowchart of an image generation method according to an exemplary embodiment, the method is executed by a server, such as Figure 5 As shown, the specific steps include:

[0140] In step 501, the server obtains a first image and a second image.

[0141] In some embodiments, the terminal provides a first upload interface and a second upload interface, the first upload interface is used to upload the first image, and the second upload interface is used to select the second image or upload the second image. The server obtains the first image and the second image based on the first upload interface and the second upload interface, respectively, including the following two processes:

[0142] Process 1: The server obtains the first image. The terminal displays the first upload interface, and the user uploads an image through the first upload interface. The terminal obtains the image uploaded by the user as the first image, and sends the first image to the server. The server receives the first image sent by the terminal.

[0143] Process 2: The server obtains the second image. In response to the terminal detecting that the user has performed a submission operation on the first upload interface, the terminal jumps to the second upload interface, and the user can select the second image or upload the second image through the second upload interface. The process is as follows:

[0144] In one implementation, in response to a user triggering an image library browsing request on a second upload interface, the terminal displays at least one image in the image library on the second upload interface, the user selects an image from at least one image displayed on the second upload interface, the selected image is the second image, the terminal sends a second image identifier to the server, the server receives the second image identifier sent by the terminal, and obtains the second image based on the second image identifier.

[0145] In another implementation, in response to a user uploading an image through the second upload interface, the terminal obtains the image uploaded by the user as a second image, sends the second image to the server, and the server receives the second image sent by the terminal.

[0146] In the above embodiment, the sequential acquisition of the first image and the second image is taken as an example for explanation, but in some embodiments, the order can also be disrupted, for example, the second image is acquired first, and then the first image is acquired. The embodiment of the present application does not limit this.

[0147] In some embodiments, the first upload interface and the second upload interface can be provided as one upload interface, that is, the user can upload the first image and the second image in any order through the upload interface, or, the upload interface can be used to upload the first image and determine the second image based on the user's selection operation. The embodiments of the present application are not limited to this.

[0148] In step 502, the server obtains images of various body parts of the first object in the first image based on the first image.

[0149] In some embodiments, this step 502 is implemented based on the following steps 502A to 502B.

[0150] In step 502A, the server inputs the first image into the semantic segmentation sub-model to obtain semantic annotation information of the first image.

[0151] In some embodiments, the server inputs the first image into a semantic segmentation sub-model, extracts features of the first image through the model, performs a corresponding semantic classification process based on the extracted features, and then outputs semantic annotation information of the first image.

[0152] Exemplarily, taking the fully convolutional neural network as the semantic segmentation submodel as an example, the above model processing process is explained: the semantic segmentation submodel includes multiple convolutional layers and at least one upsampling unit. Through the multiple convolutional layers, the first image is subjected to multiple convolution operations to obtain feature maps of multiple scales. For a feature map of any scale, the semantic segmentation submodel maps the feature map of the scale to the semantic annotation space, obtains the probability that each pixel in the feature map of the scale belongs to each semantic category, and the semantic category refers to each part of the body of the first object, such as hair, face, top, arm, etc., determines the semantic category corresponding to the maximum probability value of each pixel as the semantic category of the pixel, obtains the semantic category of each pixel in the feature map of the scale, and the semantic category of each pixel in the feature map of the scale is the semantic annotation information of the feature map of the scale. Through at least one upsampling unit, the semantic annotation information of the feature maps of each scale is upsampled respectively, and the semantic annotation information of each upsampled feature map is fused to obtain the semantic annotation information of the first image.

[0153] In step 502B, the server obtains images of various body parts of the first object in the first image based on the semantic annotation information of the first image.

[0154] In some embodiments, this step 502B is implemented based on the following two processes.

[0155] Process 1: Generate a mask: The server generates a mask of each body part of the first object based on the semantic annotation information of the first image.

[0156] Exemplarily, for any body part of the first object, the process of generating a mask includes: based on the size of the first image, generating a matrix of the same size as the first image, setting the elements of the matrix corresponding to the pixel positions of the body part in the first image to 1, and setting the elements of the remaining positions in the matrix to 0, and the matrix is ​​the mask of the body part.

[0157] Process 2: Obtaining images of various body parts: The server performs dot multiplication of the first image with the masks of various body parts of the first object respectively to obtain images of various body parts of the first object in the first image.

[0158] It should be noted that the masks of the various body parts of the above-mentioned first object are all of the same scale as that of the first image, and the images of the various body parts of the above-mentioned first object are obtained by performing dot product of the first image and the masks of the various body parts of the first object. Therefore, the images of the various body parts of the first object are all of the same scale as that of the first image.

[0159] In step 503, the server obtains a plurality of first features based on the images of the body parts of the first object, the plurality of first features including features corresponding to the plurality of body parts of the first object in the first image, and the features corresponding to each body part of the first object include features of multiple scales.

[0160] In some embodiments, the server inputs each body part image of the first object into the feature extraction sub-model respectively, and for any body part image of the first object, obtains first features of multiple scales output by multiple network layers in the feature extraction sub-model, and for the first feature of any scale except the smallest scale, the first feature of the scale is not only the output data of the corresponding network layer, but also the input data of the next level network layer.

[0161] In some embodiments, for any body part image of the first object, the process of obtaining first features of multiple scales output by multiple network layers in the feature extraction submodel includes steps 503A to 503C:

[0162] In step 503A, the body part image is input into the first-level network layer of the feature extraction sub-model to obtain the first feature of the first scale output by the first-level network layer, and the first feature of the first scale is used as input data of the next-level network layer.

[0163] In step 503B, for the first feature of any scale output by any level of network layer, the first feature of the scale is input into the next level of network layer to obtain the first feature of the next level of scale.

[0164] In step 503C, in response to the scale of the first feature of the scale being the smallest scale except the smallest scale, the first feature of the scale is input into the last network layer to obtain the first feature with the smallest scale.

[0165] Taking a feature extraction sub-model including three network layers as an example, the above process is briefly introduced: the body part image P1 is input into the first-level network layer of the feature extraction sub-model, and the first feature PF1 of the first scale output by the first-level network layer is obtained; the first feature PF1 of the first scale is used as the input data of the second-level network layer, and the first feature PF1 is processed by the second-level network layer to output the first feature PF2 of the second scale; PF2 is used as the input data of the third-level network layer, and the first feature PF2 is processed by the third-level network layer to output the first feature PF3 of the third scale. At this time, the output of the feature extraction sub-model is the first features PF1, PF2 and PF3 of three scales.

[0166] In some embodiments, for any body part image of the first object, such as Figure 6 As shown, the feature extraction sub-model introduces human semantic features of multiple scales to realize the process of extracting first features of multiple scales. The human semantic features of multiple scales are features extracted based on the common objects in context (COCO) dataset. The process is completed by steps 503D to 503G:

[0167] In step 503D, the server obtains human semantic features of multiple scales of the body part from the database.

[0168] In step 503E, the body part image of the first object is input into the first-level network layer in the feature extraction submodel to obtain the first feature of the first scale output by the first-level network layer, and the first feature of the first scale and the human body semantic feature of the first scale are used as input data of the next-level network layer.

[0169] In step 503F, for the first feature of any scale output by any level of network layer, the first feature of the scale is concatenated with the human body semantic feature of the same scale, and the concatenated first feature of the scale is input into the next level of network layer to obtain the first feature of the next level of scale.

[0170] In step 503G, in response to the scale of the first feature of the scale being the smallest scale except the minimum scale, the first feature of the scale is concatenated with the human body semantic feature of the same scale, and the concatenated first feature of the scale is input into the last network layer to obtain the first feature with the smallest scale.

[0171] Based on the above example of a feature extraction sub-model including three network layers, in the solution provided by steps 503D to 503G, the output data of each level of network layer will be spliced ​​with the human body semantic features of the corresponding scale, and then input into the next level of network layer, so as to obtain more accurate and representative first features of multiple scales through processing of each level of network layers.

[0172] The solution proposed in the embodiment of the present disclosure, by introducing the features of the COCO dataset, can not only guide the extraction of human features and provide some low-level features, but also avoid the overfitting phenomenon caused by extracting features based on limited samples.

[0173] In step 504, the server obtains optical flow information between the second image and the first image, where the optical flow information is used to represent deformation information of the same pixel in different images.

[0174] In some embodiments, the above step 504 is implemented based on the following steps 504A to 504B.

[0175] In step 504A, the server obtains masks of various body parts of the second object in the second image based on the second image.

[0176] In some embodiments, if the second image is uploaded by the user, the server obtains masks of various body parts of the second object in the second image based on the same processing as step 502. If the second image is selected by the user based on the image library provided by the server, the server obtains the semantic annotation information corresponding to the second image from the image library, and obtains masks of various body parts of the second object in the second image based on the same processing as step 502B.

[0177] It is understandable that the process of acquiring the mask corresponding to the second image in step 504A can be performed at any time after the second image is acquired, as long as it is completed before the following step 504B is required, and is not limited to the current timing.

[0178] In step 504B, the server obtains optical flow information corresponding to each body part based on the mask of each body part of the second object and the mask of each body part of the first object.

[0179] Step 1: Rescale the mask.

[0180] In some embodiments, the server scales the masks of each body part of the second object and the masks of each body part of the first object according to the multiple scales, to obtain masks of each body part of the second object at multiple scales and masks of each body part of the first object at multiple scales.

[0181] Process 2: Obtain multiple optical flow information.

[0182] In some embodiments, the server obtains optical flow information of multiple scales corresponding to each body part based on masks of each body part of the second object and masks of each body part of the first object at the same scale.

[0183] Exemplarily, the server inputs multiple groups of masks of various body parts of the second object and various body parts of the first object of the same scale into the optical flow deformation sub-model respectively to obtain multiple optical flow information, wherein the optical flow information includes optical flow data and trusted area data, wherein the optical flow data is the movement distance of the same pixel in the masks of various body parts of the second object and various body parts of the first object, and the trusted area data is the credibility of the optical flow data of each pixel.

[0184] For any set of masks of the body parts of the second object and the body parts of the first object at the same scale, the process of obtaining the optical flow information of the scale can be expressed by formula (1):

[0185]

[0186] Among them, F ifw () represents the optical flow deformation sub-model, the input data of which is the mask of each body part of the first object at this scale and the masks of the body parts of the second object at that scale N represents the number of body parts, represents the mask of the i-th body part of the first object at this scale, The output data of the optical flow deformation sub-model is the optical flow data of this scale. and the credible area data of this scale w i Represents the optical flow data of the i-th body part at this scale, a i Represents the trustworthy region data of the i-th body part at this scale. Optical flow data w i The specific form of can be expressed by formula (2):

[0187]

[0188] Among them, R and C are the height and width of the i-th body part mask of the first object at this scale, is the moving distance in the x direction and the moving distance in the y direction between the pixel at the (x, y) coordinates of the ith body part mask of the first object at the scale and the corresponding pixel of the ith body part mask of the second object at the scale, and ∑ represents the traversal operation.

[0189] The solution proposed in the embodiment of the present disclosure models the deformation relationship between the first image and the second image through the optical flow deformation sub-model, and obtains the deformation information between the first object and the second object of different scales, thereby realizing the deformation of the first features of multiple scales.

[0190] In some embodiments, for this step 504, the server can perform three-dimensional modeling on the first object and the second object to obtain a first human body model and a second human body model, and obtain optical flow information between the first image and the second image based on the first human body model and the second human body model. Compared with the mask obtained based on the segmentation result, the human body three-dimensional model has depth information and can better model the occlusion relationship of the human body, thereby making the obtained optical flow information more accurate. The change of the posture of the first object based on the optical flow information can make the generated target image higher in quality.

[0191] In step 505, the server deforms the multiple first features based on the optical flow information to obtain multiple second features.

[0192] In some embodiments, the multiple optical flow information includes multiple optical flow data and multiple confidence regions, the multiple first features are multiple two-dimensional feature maps, and the server deforms the multiple first features based on the multiple optical flow information, which means that based on the movement distance of each pixel point in the multiple optical flow data, each pixel point in the corresponding multiple two-dimensional feature maps is moved accordingly, and the multiple two-dimensional feature maps after the pixel point movement are respectively multiplied by the corresponding multiple confidence regions to obtain multiple second features. The following takes the deformation of the first feature of the clothing part as an example for explanation, see Figure 7 , Figure 7 is a schematic diagram showing a basic principle of optical flow deformation according to an exemplary embodiment. For a mask of a clothing part of a first object and a mask of a clothing part of a second object at any scale, the server obtains the optical flow data w of the scale based on the method of step 504 above. i and the confidence region data a of this scale i , through w i and a i The second feature of the scale corresponding to the clothing part is obtained.

[0193] The solution proposed in the disclosed embodiment deforms the features based on the optical flow information. The deformation of the features extracted from the image can be regarded as an implicit deformation. Unlike the explicit deformation that directly deforms the original image, the implicit deformation can alleviate the problem of difficulty in finding corresponding points when the posture changes greatly. This solution deforms the first features of multiple scales of each body part simultaneously through multiple optical flow information, so that the features of different scales can be effectively used for posture transformation in the subsequent image generation process. At the same time, by multiplying the deformed features with the confidence region, the area with low confidence in the optical flow data is filtered out, thereby improving the quality of the generated target image.

[0194] In step 506, the server processes the multiple second features to obtain a target image.

[0195] In some embodiments, the server inputs the plurality of second features into the texture normalization sub-model to obtain a target image, and the above process can be implemented based on steps 506A to 506D.

[0196] In step 506A, the server calls the first convolutional neural network of the texture normalization sub-model to fuse the second features of the same scale to obtain fused features of multiple scales.

[0197] In step 506B, the server performs texture normalization processing on the fused feature with the smallest scale to obtain a texture normalized feature with the smallest scale, and performs convolution processing on the normalized feature with the smallest scale.

[0198] In step 506C, for the fused features of any scale except the smallest scale, the server concatenates the fused features of the scale with the texture normalized features of the previous scale that have been convolved to obtain the concatenated features of the scale, and performs texture normalization processing on the concatenated features of the scale to obtain the texture normalized features of the scale.

[0199] In step 506D, in response to the fact that the scale of the texture normalized features of this scale is the largest, convolution processing is performed on the normalized features with the largest scale to obtain a target image.

[0200] Taking the second features of the three scales as an example, the above process is briefly introduced: the server calls the first convolutional neural network of the feature normalization sub-model, fuses the second features of the same scale, obtains the fused features of the first-level scale, the fused features of the second-level scale and the fused features of the third-level scale, performs texture normalization processing on the fused features of the first-level scale to obtain the texture normalization features of the first-level scale, performs convolution processing on the texture normalization features of the first-level scale, and combines the texture normalization features of the first-level scale that have been convolved with the fused features of the second-level scale. The features are spliced ​​to obtain splicing features of the second scale, the splicing features of the second scale are texture normalized to obtain texture normalized features of the second scale, the texture normalized features of the second scale are convolved, the convolved texture normalized features of the second scale are spliced ​​with the fusion features of the third scale to obtain splicing features of the third scale, the splicing features of the third scale are texture normalized to obtain texture normalized features of the third scale, the texture normalized features of the third scale are convolved to obtain the target image.

[0201] Exemplarily, the process of obtaining the texture normalization feature is described. The server scales the image of the clothing part in the first image to obtain clothing part images of multiple scales. The second convolutional neural network based on the texture normalization sub-model extracts features from the clothing part images of multiple scales to obtain texture features of multiple scales. For features corresponding to any scale, such as Figure 8 As shown, the server normalizes the features of the scale based on the mean and variance of each element in the features of the scale to obtain the normalized features of the scale, and fuses the texture features of the scale with the normalized features of the scale to obtain the texture normalized features of the scale. The process of obtaining the texture normalized features is shown in formula (3).

[0202]

[0203] Among them, h represents the smallest scale fusion feature or the concatenated feature of any scale except the smallest scale, μ() represents the mean operation, and Wan() represents the variance operation. represents the normalized features of any scale, γ cloth and β cloth represents the texture feature at any scale, h s Represents the normalized features of texture at any scale.

[0204] For example, taking the example of the server obtaining the minimum-scale texture normalization feature, the server scales the clothing part image in the first image to obtain the minimum-scale clothing part image, and extracts features of the minimum-scale clothing part image based on the second convolutional neural network of the texture normalization sub-model to obtain the minimum-scale texture feature. Based on the above formula (3), the minimum-scale fusion feature is normalized to obtain the minimum-scale normalized feature, and the minimum-scale normalized feature and the minimum-scale texture feature are fused to obtain the minimum-scale texture normalized feature.

[0205] By performing texture normalization operation on the features, the texture in the first image is fitted into the target image, so that the generated target image can maintain texture details, thereby achieving the purpose of improving the quality of the target image.

[0206] The technical solution provided by the embodiment of the present disclosure first obtains features of multiple scales in the first image and the second image for indicating the body part of the object, and then deforms multiple first features of the first image based on the optical flow information of multiple scales between the first image and the second image to achieve the purpose of changing the posture of the first object, and then generates an image based on the deformed features. Since the deformation in the above-mentioned image generation process is performed while retaining the complete feature information of the image, the accuracy of the texture in the generated target image is guaranteed when a large posture change is performed, thereby improving the quality of the generated target image.

[0207] Above Figure 5 In the corresponding embodiment, a method for obtaining a target image by changing the posture of an object in an input image is introduced. The image generation method provided in this embodiment is based on Figure 4 The image generation model shown in the figure changes the posture and body parts of the object in the input image to obtain the target image. Fig. 9 This method is described. Fig. 9 is a flowchart of an image generation method according to an exemplary embodiment, the method is executed by a server, such as Fig. 9 As shown, the specific steps include:

[0208] In step 901, the server obtains a first image, a second image, and a third image.

[0209] The third image and the second image are the same image; or the third image and the second image are different images.

[0210] In some embodiments, the terminal provides a first upload interface, a second upload interface, and a third upload interface, the first upload interface is used to upload the first image, the second upload interface is used to select the second image or upload the second image, and the third upload interface is used to select the third image or upload the third image. The server obtains the first image, the second image, and the third image based on the first upload interface, the second upload interface, and the third upload interface, respectively, including the following three processes:

[0211] Process 1: The server obtains the first image. This process 1 is the same as the process 1 in the above step 501, and will not be described again.

[0212] Process 2: The server obtains the second image. Process 2 is the same as process 2 in step 501 above, and will not be described again.

[0213] Process 3: The server obtains the third image. In response to the terminal detecting that the user has performed a submission operation on the second upload interface, the terminal jumps to the third upload interface, and the user can select the second image as the third image through the third upload interface, that is, the second image and the third image are the same image, or the user can select or upload a third image different from the second image through the third upload interface, and the process is as follows:

[0214] In one implementation, the second image and the third image are the same image. In response to the user selecting the second image as the third image on the third upload interface, the terminal displays options for various body parts on the third upload interface. The user selects a body part from the options displayed on the third upload interface. The selected body part is the target body part. The terminal sends a target body part identifier and an indication message to the server, where the indication message is used to indicate that the second image and the third image are the same image. The server receives the indication message and target body part identifier sent by the terminal.

[0215] In another implementation, the second image and the third image are different images. In response to the user triggering an image library browsing request on the third upload interface, the terminal displays at least one image in the image library on the third upload interface. The image library is an image library preset by the server. The image library includes at least one image, each image corresponds to an image identifier and semantic annotation information of the image. The user selects an image from at least one image displayed on the third upload interface. In response to the terminal's selection operation on the image, options for various body parts are displayed on the third upload interface. The user selects a body part from the options displayed on the third upload interface. The selected image is the third image, and the selected body part is the target body part. The terminal sends the third image identifier and the target body part identifier to the server. The server receives the third image identifier and the target body part identifier sent by the terminal, and obtains the third image based on the third image identifier.

[0216] In another implementation, the second image and the third image are different images. In response to the user uploading an image through a third image upload interface, the terminal obtains the image uploaded by the user as the third image. In response to the terminal detecting that the user has completed uploading the image, options for various body parts are displayed on the third upload interface. The user selects a body part from the options displayed on the third upload interface. The selected body part is the target body part. The terminal sends the third image and the target body part identifier to the server, and the server receives the third image and the target body part identifier sent by the terminal.

[0217] In the above embodiment, acquiring the second image and the third image needs to be performed sequentially, and acquiring the first image can be performed before acquiring the second image, or after acquiring the third image, which is not limited in the embodiment of the present application.

[0218] In some embodiments, the above-mentioned first upload interface, second upload interface and third upload interface can be provided as one upload interface, that is, the user can first upload the first image through the upload interface, and then upload or select the second image and the third image; alternatively, the user can first upload or select the second image and the third image through the upload interface, and then upload the first image. The embodiments of the present application are not limited to this.

[0219] In step 902, the server obtains images of various body parts of the first object in the first image based on the first image.

[0220] In some embodiments, the server obtains images of various body parts of the first object in the first image based on the same method as the above step 502, which will not be described in detail here.

[0221] In step 903, the server obtains a plurality of first features based on the images of the body parts of the first object, the plurality of first features including features corresponding to the plurality of body parts of the first object in the first image, and the features corresponding to each body part of the first object include features of multiple scales.

[0222] In some embodiments, the server obtains the multiple first features based on the same method as step 503 above, which will not be repeated here.

[0223] In step 904, the server obtains optical flow information between the second image and the first image, where the optical flow information is used to represent deformation information of the same pixel in different images.

[0224] In some embodiments, the server obtains the optical flow information based on the same method as step 504 above, which will not be described in detail here.

[0225] In step 905, the server obtains images of various body parts of the third object in the third image based on the third image.

[0226] In some embodiments, the server obtains images of various body parts of the third object based on the same method as the above step 502, which will not be described in detail here.

[0227] In step 906, the server obtains multiple third features of the third image based on the images of various body parts of the third object, the multiple third features include features corresponding to multiple body parts of the third object in the third image, the features corresponding to each body part of the third object include features of multiple scales, and the third object in the third image includes a target body part.

[0228] In some embodiments, the server obtains the multiple third features based on the same method as the above step 503, which will not be repeated here.

[0229] In step 907, the server replaces the first feature corresponding to the target body part in the multiple first features based on the third feature corresponding to the target body part in the multiple third features, where the target body part includes one or more body parts of the third object.

[0230] In some embodiments, the server determines the third features of multiple scales corresponding to the target body part based on the identification of the target body part, and determines the first features of multiple scales corresponding to the target body part from the multiple first features, and replaces the first features of the corresponding scales corresponding to the target body part with the third features of the multiple scales.

[0231] For example, Fig.10 As shown, Fig.10 The present invention is a schematic diagram of a basic principle of feature replacement according to an exemplary embodiment. The framed portion of the third image in the figure is a target body part. The first feature of the corresponding body part of the first object in the first image is replaced by the third feature of the target body part, thereby achieving a change in the body part of the first object in the first image.

[0232] The solution proposed in the embodiment of the present disclosure can replace the features of the body part that is expected to be generated with the first features, thereby achieving changes to the body part in the first object.

[0233] In step 908, the server deforms the multiple replaced first features and other non-replaced first features based on the optical flow information to obtain multiple second features.

[0234] In some embodiments, the server deforms the multiple replaced first features and other non-replaced first features based on the same method as the above step 505 to obtain multiple second features, which will not be repeated here.

[0235] In step 909, the server processes the multiple second features to obtain a target image.

[0236] In some embodiments, the server processes the multiple second features based on the same method as the above step 506 to obtain the target image, which will not be repeated here.

[0237] It should be noted that, for this step 909, the server determines to perform texture normalization processing based on the image of the clothes part in the first image or the image of the clothes part in the third image based on the target body part identification obtained in the above step 901; in response to the target body part identification indicating that the target body part is a clothes part, the server performs texture normalization processing based on the image of the clothes part in the third image; in response to the target body part identification indicating that the target body part is not a clothes part, the server performs texture normalization processing based on the image of the clothes part in the first image.

[0238] Fig.11 is a comparison diagram of the results of the present technical solution and the ADGAN method according to an exemplary embodiment. Fig.11 As shown in the figure, it includes multiple first images, multiple third images, multiple target posture images, multiple target images generated by the ADGAN method, and multiple target images generated by the present technical solution. Among them, the framed part in the third image is the target body part, and the target posture image includes multiple position points. The multiple position points are connected to form an abstract posture. The ADGAN method performs a posture transformation on the first object in the first image based on the target posture image. In the present technical solution, the server can obtain a second image containing the same posture based on the target posture image. Based on Fig.11 The comparison results shown show that the texture in the target image generated by this technical solution is more accurate and the quality of the target image is higher.

[0239] To further demonstrate the effect of the target image generated by this technical solution, see Fig.12 , Fig.12 is a result diagram of an image generation method according to an exemplary embodiment. Fig.12 The displayed target image is obtained by deforming the body part and posture of the first object in the first image based on the same image, that is, the second image and the third image are the same image. Fig.12From the displayed target image, it can be seen that the technical solution can achieve changes in body parts and postures of the object in the first image, and ensure the accuracy of the texture in the generated target image.

[0240] The technical solution provided by the disclosed embodiment obtains features of multiple scales in the first image and the third image for indicating the body parts of the object, and changes the body parts based on the features of the multiple scales, and then deforms the replaced multiple first features based on the optical flow information of multiple scales between the first image and the second image, so as to achieve the purpose of changing the posture of the first object, and then generates an image based on the deformed features. Since the deformation in the above-mentioned image generation process is performed while retaining the complete feature information of the image, the accuracy of the texture in the generated target image is guaranteed when a large posture change is performed, thereby improving the quality of the generated target image.

[0241] In some embodiments, the server can process the first image based on the reference video and generate a target video based on the processed image. The method provided in the present application also includes: the server obtains at least one image from the reference video, uses the image as the second image, changes the body part and posture of the first object in the first image based on the at least one second image to obtain at least one target image, and generates a target video based on the target image corresponding to the at least one image.

[0242] Exemplarily, the process of generating a target video based on a target image is described. The at least one image is a continuous video frame in a reference video, and the server splices the corresponding at least one target image together in the same order based on the order of the at least one image in the reference video to obtain the target video.

[0243] The following is an introduction to the training process of the image generation model involved in the above embodiment: the training process includes two parts. First, the semantic segmentation sub-model is pre-trained, and then, based on the pre-trained semantic segmentation sub-model, the image generation model is trained as a whole. The specific instructions are as follows.

[0244] (1) The server pre-trains the semantic segmentation sub-model based on the public look into person (LIP) dataset, which includes sample human images and sample semantic annotation information.

[0245] (2) During the overall training of the image generation model, the server obtains training data, which includes a first sample image, a second sample image, a third sample image, and a sample target image. The training is implemented through multiple iterations. During any iteration, the server obtains the masks of the various body parts of the object in the second sample image, inputs the masks of the various body parts of the object in the first sample image, the second sample image, and the third sample image into the model to be trained, and obtains a sample generated image. Based on the output sample generated image and sample target image, it is determined whether the training end condition is met. If so, the model corresponding to this iteration is determined as the above-mentioned image generation model. If not, the model parameters are adjusted, and the next iteration process is performed based on the adjusted model. Optionally, the above-mentioned training end condition is: if the difference value between the sample generated image and the sample target image is less than 0.01 or the number of iterations is greater than the first threshold, then the training is terminated.

[0246] It should be noted that, in any generation process of the overall training of the image generation model, the server obtains multiple sample optical flow information based on the first sample image, the second sample image and the optical flow deformation sub-model, and the sample optical flow information includes sample optical flow data and sample trusted region data. The specific acquisition process is as described in step 504 and will not be repeated here. The server scales the first sample image and the second sample image to obtain the first sample image and the second sample image of multiple scales. For any group of first sample images and second sample images of the same scale, the loss of the optical flow deformation sub-model is calculated based on the self-supervised loss function, as shown in formula (4):

[0247]

[0248] Among them, Loss flow represents the loss of the optical flow deformation sub-model, I s represents the first sample image of this scale, Represents the mask of the i-th body part of the object in the first sample image of this scale, I t represents the second sample image of this scale, represents the mask of the i-th body part of the object in the third sample image of this scale, a i represents the sample credible region data of the i-th body part at this scale, The calculation process of is shown in formula (5).

[0249]

[0250] Among them, I s represents the first sample image of the scale, x and y represent the coordinates of any pixel in the first sample image of the scale, and The optical flow information of the pixel with coordinates (x, y) in the image of the i-th body part of the object in the first sample image of the scale is represented. Formula (5) represents the corresponding movement of each pixel in the image of the i-th body part of the object in the first sample image of the scale based on the sample optical flow data of the corresponding scale. Through the optical flow self-supervised loss function, the optical flow deformation sub-model can achieve adaptive optimization in the training process of multiple iterations.

[0251] Through the above training process, the model can change the body part and posture of the first object in the first image based on the input first image, second image and third image, and generate a high-quality target image.

[0252] Fig.13 is a block diagram of an image generating device according to an exemplary embodiment. Fig.13 The device includes: an acquisition unit 1301, a deformation unit 1302 and a processing unit 1303.

[0253] An acquiring unit 1301 is configured to acquire a plurality of first features of a first image, wherein the plurality of first features include features corresponding to a plurality of body parts of a first object in the first image, and the features corresponding to each body part of the first object include features of a plurality of scales;

[0254] The acquisition unit 1301 is configured to acquire a second image, where the second image is used to provide a target posture of the first object;

[0255] The acquiring unit 1301 is configured to acquire optical flow information between the second image and the first image, where the optical flow information is used to represent deformation information of the same pixel in different images;

[0256] A deformation unit 1302 is configured to perform deformation on the plurality of first features based on the optical flow information to obtain a plurality of second features;

[0257] The processing unit 1303 is configured to process the multiple second features to obtain a target image.

[0258] In some embodiments, the acquisition unit 1301 is configured to acquire, based on the first image, images of various body parts of the first object in the first image; and acquire the multiple first features based on the images of various body parts of the first object.

[0259] In some embodiments, the acquisition unit 1301 includes:

[0260] The input subunit is configured to input each body part image of the first object into the feature extraction submodel respectively, and for any body part image of the first object, obtain first features of multiple scales output by multiple network layers in the feature extraction submodel.

[0261] In some embodiments, the input subunit is configured to execute the acquisition of human semantic features of multiple scales of the body part, where the human semantic features of multiple scales are features extracted based on a general object data set; input the image of the body part of the first object into the first-level network layer in the feature extraction submodel to obtain the first feature of the first scale output by the first-level network layer, and use the first feature of the first scale and the human semantic features of the first scale as input data of the next-level network layer; for the first feature of any scale output by any-level network layer, splice the first feature of the scale with the human semantic features of the same scale, and input the spliced ​​first feature of the scale into the next-level network layer to obtain the first feature of the next-level scale; in response to the scale of the first feature of the scale being the smallest scale except the smallest scale, splice the first feature of the scale with the human semantic features of the same scale, and input the spliced ​​first feature of the scale into the last network layer to obtain the first feature of the smallest scale.

[0262] In some embodiments, the deformation unit 1302 includes:

[0263] an acquisition subunit, configured to acquire masks of various body parts of the second object in the second image based on the second image;

[0264] The acquisition subunit is configured to execute, based on the masks of the respective body parts of the second object and the masks of the respective body parts of the first object, to acquire the optical flow information corresponding to the respective body parts;

[0265] The deformation subunit is configured to perform deformation on the multiple first features based on the optical flow information corresponding to each body part to obtain the multiple second features.

[0266] In some embodiments, the acquisition subunit is configured to perform scale changes on the masks of each body part of the second object and the masks of each body part of the first object according to the multiple scales, respectively, to obtain masks of each body part of the second object at the multiple scales and masks of each body part of the first object at the multiple scales; based on the masks of each body part of the second object and the masks of each body part of the first object at the same scale, obtain optical flow information of multiple scales corresponding to the each body part.

[0267] In some embodiments, the processing unit 1303 includes:

[0268] A fusion subunit is configured to fuse the second features of the same scale to obtain fused features of multiple scales;

[0269] The normalization subunit is configured to perform texture normalization processing on the fused feature with the smallest scale to obtain the texture normalized feature with the smallest scale, and perform convolution processing on the texture normalized feature with the smallest scale;

[0270] The splicing subunit is configured to execute, for the fusion features of any scale except the smallest scale, splicing the fusion features of the scale with the texture normalization features of the previous scale that have been processed by convolution to obtain the splicing features of the scale, and performing texture normalization processing on the splicing features of the scale to obtain the texture normalization features of the scale;

[0271] The processing subunit is configured to execute, in response to the texture normalization feature of the scale being the texture normalization feature of the largest scale, performing convolution processing on the texture normalization feature of the largest scale to obtain a target image.

[0272] In some embodiments, the apparatus further comprises:

[0273] a scale change unit configured to perform scale change on the image of the clothing part in the first image to obtain the clothing part images of the plurality of scales; and perform feature extraction on the clothing part images of the plurality of scales to obtain texture features of the plurality of scales;

[0274] The normalization subunit is configured to perform normalization processing on the features of the scale based on the mean and variance of each element in the features of the scale to obtain the normalized features of the scale; and to fuse the texture features of the scale with the normalized features of the scale to obtain the texture normalized features of the scale.

[0275] In some embodiments, the acquisition unit 1301 is configured to acquire a plurality of third features of a third image, the plurality of third features including features corresponding to a plurality of body parts of a third object in the third image, the features corresponding to each body part of the third object including features of the plurality of scales, and the third object in the third image including a target body part;

[0276] The device also includes:

[0277] A replacement unit is configured to replace a first feature corresponding to the target body part among the multiple first features based on a third feature corresponding to the target body part among the multiple third features, where the target body part includes one or more body parts of the third object.

[0278] In some embodiments, the third image and the second image are the same image; or, the third image and the second image are different images.

[0279] In some embodiments, the acquisition unit 1301 is configured to acquire at least one image from a reference video and use the image as the second image;

[0280] The device also includes:

[0281] The generating unit is configured to generate a target video based on the target image corresponding to the at least one image.

[0282] It should be noted that: the image generation device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate the image generation. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the image generation device provided in the above embodiment and the image generation method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0283] The above embodiment is described by taking the electronic device as a server as an example. The structure of the electronic device is described below. Fig.14 It is a block diagram of an electronic device according to an exemplary embodiment. The electronic device 1400 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 1401 and one or more memories 1402, wherein the one or more memories 1402 store at least one program code, and the at least one program code is loaded and executed by the one or more processors 1401 to implement the process performed by the electronic device in the image generation method provided by the above-mentioned various method embodiments. Of course, the electronic device 1400 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 1400 may also include other components for implementing device functions, which will not be repeated here.

[0284] In an exemplary embodiment, a computer-readable storage medium including a program code is also provided, such as a memory 1402 including the program code, and the program code can be executed by a processor 1401 of an electronic device 1400 to complete the above-mentioned image generation method. Optionally, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact-disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0285] In an exemplary embodiment, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned image generating method is implemented.

[0286] In some embodiments, the computer program involved in the embodiments of the present application may be deployed and executed on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed at multiple locations and interconnected by a communication network. Multiple electronic devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain system.

[0287] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0288] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire a plurality of first features of the first image, the plurality of first features comprising features corresponding to a plurality of body parts of a first object in the first image, wherein the features corresponding to each body part of the first object comprise features of a plurality of scales; Acquire a second image, where the second image is used to provide a target posture of the first object; Acquire optical flow information between the second image and the first image, where the optical flow information is used to represent deformation information of the same pixel in different images; Based on the optical flow information, deforming the plurality of first features to obtain a plurality of second features; The second features of the same scale are fused to obtain fused features of multiple scales; Perform texture normalization processing on the smallest scale fusion feature to obtain the smallest scale texture normalization feature, and perform convolution processing on the smallest scale texture normalization feature; For the fusion features of any scale except the smallest scale, the fusion features of the scale are concatenated with the texture normalization features of the previous scale after convolution processing to obtain the concatenated features of the scale, and the texture normalization processing is performed on the concatenated features of the scale to obtain the texture normalization features of the scale; In response to the texture normalization feature of the scale being the texture normalization feature of the largest scale, convolution processing is performed on the texture normalization feature of the largest scale to obtain a target image.

2. The image generation method according to claim 1, characterized in that: The acquiring of a plurality of first features of the first image comprises: Based on the first image, acquiring images of various body parts of the first object in the first image; The plurality of first features are acquired based on images of respective body parts of the first object.

3. The image generation method according to claim 2, characterized in that: The acquiring the plurality of first features based on the images of the respective body parts of the first object comprises: The images of each body part of the first object are respectively input into the feature extraction sub-model, and for any body part image of the first object, first features of multiple scales output by multiple network layers in the feature extraction sub-model are obtained.

4. The image generation method according to claim 3, characterized in that: For any body part image of the first object, obtaining first features of multiple scales output by multiple network layers in the feature extraction submodel comprises: Acquire human semantic features of multiple scales of the body part, where the human semantic features of multiple scales are features extracted based on a general object dataset; Inputting the body part image of the first object into the first-level network layer in the feature extraction sub-model, obtaining the first feature of the first scale output by the first-level network layer, and using the first feature of the first scale and the human body semantic feature of the first scale as input data of the next-level network layer; For the first feature of any scale output by any level of network layer, the first feature of the scale is concatenated with the human body semantic feature of the same scale, and the concatenated first feature of the scale is input into the next level of network layer to obtain the first feature of the next level of scale; In response to the scale of the first feature of the scale being the smallest scale except the smallest scale, the first feature of the scale is spliced ​​with a human body semantic feature of the same scale, and the spliced ​​first feature of the scale is input into the last network layer to obtain the first feature with the smallest scale.

5. The image generation method according to claim 1, characterized in that: The deforming the plurality of first features based on the optical flow information to obtain the plurality of second features comprises: Based on the second image, obtaining masks of various body parts of the second object in the second image; Based on the masks of the respective body parts of the second object and the masks of the respective body parts of the first object, acquiring optical flow information corresponding to the respective body parts; Based on the optical flow information corresponding to each body part, the multiple first features are deformed to obtain the multiple second features.

6. The image generation method according to claim 5, characterized in that: The acquiring optical flow information of each body part based on the mask of each body part of the second object and the mask of each body part of the first object comprises: According to the multiple scales, respectively scale the masks of the respective body parts of the second object and the masks of the respective body parts of the first object to obtain the masks of the respective body parts of the second object at the multiple scales and the masks of the respective body parts of the first object at the multiple scales; Based on the masks of the respective body parts of the second object and the masks of the respective body parts of the first object at the same scale, optical flow information of multiple scales corresponding to the respective body parts is obtained.

7. The image generation method according to claim 1, characterized in that: The method further comprises: Performing a scale change on the image of the clothing part in the first image to obtain images of the clothing part at multiple scales; Extracting features from the clothing part images at the multiple scales to obtain texture features at the multiple scales; The texture normalization processing for features corresponding to any scale includes: Based on the mean and variance of each element in the feature of the scale, normalizing the feature of the scale to obtain the normalized feature of the scale; The texture features of the scale are fused with the normalized features of the scale to obtain the texture normalized features of the scale.

8. The image generation method according to claim 1, characterized in that: Before deforming the plurality of first features based on the optical flow information to obtain the plurality of second features, the method further includes: Acquire a plurality of third features of a third image, the plurality of third features comprising features corresponding to a plurality of body parts of a third object in the third image, the features corresponding to each body part of the third object comprising features of the plurality of scales, and the third object in the third image comprising a target body part; Based on the third feature corresponding to the target body part in the plurality of third features, the first feature corresponding to the target body part in the plurality of first features is replaced, and the target body part includes one or more body parts of the third object.

9. The image generation method according to claim 8, characterized in that: The third image and the second image are the same image; or, the third image and the second image are different images.

10. The image generation method according to any one of claims 1 to 9, characterized in that: The acquiring of the second image comprises: Acquire at least one image from a reference video, and use the image as the second image; After the plurality of second features are processed to obtain a target image, the method further includes: A target video is generated based on the target image corresponding to the at least one image.

11. An image generating device, characterized in that: The device comprises: an acquisition unit configured to acquire a plurality of first features of the first image, wherein the plurality of first features include features corresponding to a plurality of body parts of a first object in the first image, and the features corresponding to each body part of the first object include features of a plurality of scales; The acquisition unit is configured to acquire a second image, where the second image is used to provide a target posture of the first object; The acquisition unit is configured to acquire optical flow information between the second image and the first image, where the optical flow information is used to represent deformation information of the same pixel in different images; A deformation unit, configured to perform deformation on the plurality of first features based on the optical flow information to obtain a plurality of second features; A processing unit, including a fusion subunit, a normalization subunit, a splicing subunit and a processing subunit; The fusion subunit is configured to fuse the second features of the same scale to obtain fused features of multiple scales; The normalization subunit is configured to perform texture normalization processing on the fusion feature with the smallest scale to obtain the texture normalization feature with the smallest scale, and perform convolution processing on the texture normalization feature with the smallest scale; The splicing subunit is configured to execute, for fusion features of any scale except the smallest scale, splicing the fusion features of the scale with the texture normalization features of the previous scale that have been subjected to convolution processing to obtain splicing features of the scale, and performing texture normalization processing on the splicing features of the scale to obtain texture normalization features of the scale; The processing subunit is configured to execute, in response to the texture normalization feature of the scale being the texture normalization feature of the largest scale, performing convolution processing on the texture normalization feature of the largest scale to obtain a target image.

12. The image generating device according to claim 11, characterized in that: The acquisition unit is configured to acquire, based on the first image, images of various body parts of the first object in the first image; and acquire the multiple first features based on the images of various body parts of the first object.

13. The image generating device according to claim 12, characterized in that: The acquisition unit comprises: The input subunit is configured to input the body part images of the first object into the feature extraction submodel respectively, and for any body part image of the first object, obtain the first features of multiple scales output by multiple network layers in the feature extraction submodel.

14. The image generating device according to claim 13, characterized in that: The input subunit is configured to execute the acquisition of human semantic features of multiple scales of the body part, wherein the human semantic features of the multiple scales are features extracted based on a general object data set; input the body part image of the first object into the first-level network layer in the feature extraction submodel, obtain the first feature of the first scale output by the first-level network layer, and use the first feature of the first scale and the human semantic features of the first scale as input data of the next-level network layer; for the first feature of any scale output by any-level network layer, splice the first feature of the scale with the human semantic features of the same scale, and input the spliced ​​first feature of the scale into the next-level network layer to obtain the first feature of the next-level scale; In response to the scale of the first feature of the scale being the smallest scale except the smallest scale, the first feature of the scale is spliced ​​with a human body semantic feature of the same scale, and the spliced ​​first feature of the scale is input into the last network layer to obtain the first feature with the smallest scale.

15. The image generating device according to claim 11, characterized in that: The deformation unit comprises: an acquisition subunit, configured to acquire, based on the second image, masks of various body parts of the second object in the second image; The acquisition subunit is configured to execute, based on the masks of the respective body parts of the second object and the masks of the respective body parts of the first object, to acquire the optical flow information corresponding to the respective body parts; The deformation subunit is configured to perform deformation on the multiple first features based on the optical flow information corresponding to the various body parts to obtain the multiple second features.

16. The image generating device according to claim 15, characterized in that: The acquisition subunit is configured to perform scale changes on the masks of the respective body parts of the second object and the masks of the respective body parts of the first object respectively according to the multiple scales, so as to obtain the masks of the respective body parts of the second object at the multiple scales and the masks of the respective body parts of the first object at the multiple scales; Based on the masks of the respective body parts of the second object and the masks of the respective body parts of the first object at the same scale, optical flow information of multiple scales corresponding to the respective body parts is obtained.

17. The image generating device according to claim 11, characterized in that: The device also includes: a scale change unit, configured to perform scale change on the image of the clothing part in the first image to obtain the clothing part images of the plurality of scales; and perform feature extraction on the clothing part images of the plurality of scales to obtain the texture features of the plurality of scales; The normalization subunit is configured to perform normalization processing on the features of the scale based on the mean and variance of each element in the features of the scale to obtain the normalized features of the scale; and to fuse the texture features of the scale with the normalized features of the scale to obtain the texture normalized features of the scale.

18. The image generating device according to claim 11, characterized in that: The acquisition unit is configured to acquire a plurality of third features of a third image, wherein the plurality of third features include features corresponding to a plurality of body parts of a third object in the third image, wherein the features corresponding to the body part of each third object include features of the plurality of scales, and the third object in the third image includes a target body part; The device also includes: A replacement unit is configured to replace a first feature corresponding to the target body part among the multiple first features based on a third feature corresponding to the target body part among the multiple third features, wherein the target body part includes one or more body parts of the third object.

19. The image generating device according to claim 18, characterized in that: The third image and the second image are the same image; or, the third image and the second image are different images.

20. The image generating device according to any one of claims 11 to 19, characterized in that: The acquisition unit is configured to acquire at least one image from a reference video and use the image as the second image; The device also includes: The generating unit is configured to generate a target video based on the target image corresponding to the at least one image.

21. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; The processor is configured to execute the program code to implement the image generating method according to any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that: When the program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to execute the image generating method according to any one of claims 1 to 10.

23. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image generating method according to any one of claims 1 to 10 is implemented.

Images