Image and data processing method and device

Abstract

The present invention discloses a method and device for image and data processing. The method comprises: obtaining source posture and texture information according to a source image; obtaining a first synthetic image according to the source image, target posture and source posture; obtaining a residual image according to the texture information and the first synthetic image; and obtaining a second synthetic image according to the first synthetic image and the residual image. The present invention solves the technical problem that the synthetic image lacks realism because the texture details are omitted when extracting features during character action migration in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology applications, and in particular to an image and data processing method and device. Background Art

[0002] As the scale of e-commerce continues to expand, the types of online products, consumption and experience methods are becoming more diversified. The traditional promotional method based on text description + static photos of products can no longer fully display the highlights of products and user experience effects. The production of professional product advertising videos not only takes a lot of time, but also usually requires the participation of professionals (such as clothing models), which brings a huge burden to individual merchants and small and medium-sized enterprises to promote their products.

[0003] The currently available human motion transfer methods often lose fine texture details such as hair and clothing patterns when extracting features, resulting in overly smooth synthesized images that lack realism. Figure 1 As shown, Figure 1 is a schematic diagram of the prior art when performing character action migration, Figure 1 It can be seen that the biggest difference between the composite image and the actual photo is that the texture (pattern) on the clothes and the character features (for example, the model's hairstyle) are not clearly displayed in the composite image.

[0004] Currently, no effective solution has been proposed to address the above-mentioned problem that the existing technology may miss texture details when extracting features during character action migration, resulting in a lack of realism in the synthesized image. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for image and data processing, so as to at least solve the technical problem that the prior art omits texture details when extracting features during character action migration, resulting in a lack of realism in the synthesized image.

[0006] According to one aspect of an embodiment of the present invention, a method for image processing is provided, which includes: obtaining source posture and texture information based on a source image; obtaining a first synthetic image based on the source image, target posture and source posture; obtaining a residual map based on the texture information and the first synthetic image; and obtaining a second synthetic image based on the first synthetic image and the residual map.

[0007] Optionally, obtaining source posture and texture information according to the source image includes: obtaining the source posture of the source image by posture estimation according to the source image; and obtaining texture information of the source image by feature extraction on the source image.

[0008] Optionally, obtaining the first synthesized image according to the source image, the target posture and the source posture includes: obtaining a content feature map according to the source image, the target posture and the source posture; and obtaining the first synthesized image according to the content feature map.

[0009] Optionally, obtaining a residual map based on the texture information and the first synthesized image includes: performing normalization processing based on the texture information encoding and the content feature map to obtain a residual map; wherein, performing normalization processing based on the texture information encoding and the content feature map to obtain a residual map includes: obtaining a residual map through deep learning based on the texture information, normalizing the content feature map, and reconstructing the residual map, wherein the residual map includes contour features and surface texture details in the source image.

[0010] Further, optionally, the contour features include: at least one of human face, animal head, limb features or object appearance features; the surface texture details include: product surface texture details, wherein the product surface texture details include: at least one of clothing texture details, accessory texture details or utensil texture details.

[0011] Optionally, obtaining the second synthesized image according to the first synthesized image and the residual image includes: superimposing the first synthesized image and the residual image to obtain the second synthesized image.

[0012] Furthermore, superimposing the first composite image and the residual image to obtain the second composite image includes: filling in the positions corresponding to the contour features and surface texture details in the first composite image according to the contour features and surface texture details in the source image in the residual image to obtain the second composite image, wherein the second composite image has the contour features and surface texture details in the source image compared to the first composite image.

[0013] According to another aspect of an embodiment of the present invention, a data processing method is also provided, which includes: receiving a source image uploaded by a user; obtaining source posture and texture information of the source image based on the source image; performing posture migration based on the source image, target posture and source posture to obtain a first composite image; performing feature enhancement processing based on the texture information to obtain a residual image, wherein the residual image includes: contour features and surface texture details in the source image; filling the contour features and surface texture details in the residual image in the first composite image to generate a second composite image; and determining the second composite image as a display image for display.

[0014] According to another aspect of an embodiment of the present invention, a method for image processing is also provided, which includes: receiving a source image uploaded by a user; generating a first composite image based on a source posture of the source image, and generating a residual image based on texture information of the source image; obtaining a second composite image based on the first composite image and the residual image; and generating an image set or video data based on the second composite image.

[0015] Optionally, the image collection or video data is used for online fitting effect display or advertising page display.

[0016] According to another aspect of an embodiment of the present invention, a method for image acquisition is also provided, which includes: uploading a source image; receiving an image set or video data returned based on the source image, wherein the image set or video data generates a first composite image and a residual image based on the source posture and texture information of the source image; acquiring a second composite image based on the first composite image and the residual image; and obtaining based on the second composite image.

[0017] According to one aspect of another embodiment of the present invention, there is also provided an image processing device, which includes: a first acquisition module, used to acquire source posture and texture information based on a source image; a first synthesis module, used to acquire a first synthetic image based on the source image, the target posture and the source posture; a second acquisition module, used to acquire a residual map based on the texture information and the first synthetic image; and a second synthesis module, used to acquire a second synthetic image based on the first synthetic image and the residual map.

[0018] According to another aspect of another embodiment of the present invention, a data processing device is also provided, which includes: a receiving module for receiving a source image uploaded by a user; an acquisition module for acquiring the source posture and texture information of the source image based on the source image; a first image processing module for performing posture migration based on the source image, the target posture and the source posture, and acquiring a first composite image; a second image processing module for performing feature enhancement processing based on the texture information, and acquiring a residual image, wherein the residual image includes: contour features and surface texture details in the source image; a synthesis module for filling the contour features and surface texture details in the residual image in the first composite image, and generating a second composite image; and a display module for determining the second composite image as a display image for display.

[0019] According to another aspect of another embodiment of the present invention, there is also provided an image processing device, which includes: a receiving module for receiving a source image uploaded by a user; a first generating module for generating a first synthetic image based on a source posture of the source image, and generating a residual image based on texture information of the source image; an acquiring module for acquiring a second synthetic image based on the first synthetic image and the residual image; and a second generating module for generating an image set or video data based on the second synthetic image.

[0020] According to another aspect of another embodiment of the present invention, there is also provided an image acquisition device, which includes: an upload module for uploading a source image; a receiving module for receiving an image set or video data returned based on the source image, wherein the image set or video data generates a first composite image and a residual image based on the source posture and texture information of the source image; a second composite image is acquired based on the first composite image and the residual image; and is obtained based on the second composite image.

[0021] According to another aspect of an embodiment of the present invention, a storage medium is provided, wherein the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above-mentioned image and data processing method.

[0022] According to one aspect of yet another embodiment of the present invention, a processor is further provided, wherein the processor is used to run a program, wherein the above-mentioned image and data processing method is executed when the program is run.

[0023] In an embodiment of the present invention, a method of multiplexing an input source image to extract texture information encoding and multiplexing a content feature map obtained based on the original image, the target posture and the source posture to provide guidance containing spatial information is adopted. By obtaining source posture and texture information based on the source image; obtaining a first synthetic image based on the source image, the target posture and the source posture; obtaining a residual map based on the texture information and the first synthetic image; and obtaining a second synthetic image based on the first synthetic image and the residual map, the purpose of improving the lost / weakened detail information in the rough synthetic image is achieved, thereby achieving the technical effect of improving the realism of the synthetic image, and further solving the technical problem that the prior art omits texture details when extracting features during character action migration, resulting in a lack of realism in the synthetic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0025] Figure 1 It is a schematic diagram of the prior art when performing character action migration;

[0026] Figure 2 It is a hardware structure block diagram of a computer terminal of an image processing method according to an embodiment of the present invention;

[0027] Figure 3 is a flowchart of an image processing method according to a first embodiment of the present invention;

[0028] Figure 4a-4c is a schematic diagram of obtaining a second synthesized image according to a source image in an image processing method according to Embodiment 1 of the present invention;

[0029] Figure 5 is a flowchart of a data processing method according to a second embodiment of the present invention;

[0030] Figure 6 is a flowchart of an image processing method according to a third embodiment of the present invention;

[0031] Figure 7is a flowchart of a method for acquiring an image according to a fourth embodiment of the present invention;

[0032] Figure 8 is a schematic diagram of an image processing apparatus according to a fifth embodiment of the present invention;

[0033] Fig. 9 is a schematic diagram of a data processing device according to a sixth embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] Technical terms involved in this application:

[0037] Character pose transfer: Based on the guidance information of a given character image and target pose, synthesize an image or video of the character in a new pose.

[0038] Example 1

[0039] According to an embodiment of the present invention, an embodiment of an image processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 2FIG. 1 is a hardware structure block diagram of a computer terminal of an image processing method according to an embodiment of the present invention. Figure 2 As shown, the computer terminal 20 may include one or more (only one is shown in the figure) processors 202 (the processor 202 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 204 for storing data, and a transmission module 206 for communication functions. It can be understood by those skilled in the art that Figure 2 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 2 More or fewer components as shown, or with Figure 2 Different configurations shown.

[0041] The memory 204 can be used to store software programs and modules of application software, such as program instructions / modules corresponding to the image processing method in the embodiment of the present invention. The processor 202 executes various functional applications and data processing by running the software programs and modules stored in the memory 204, that is, the image processing method of the above-mentioned application is realized. The memory 204 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 204 may further include a memory remotely arranged relative to the processor 202, and these remote memories may be connected to the computer terminal 20 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0042] The transmission device 206 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 20. In one example, the transmission device 206 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 206 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0043] Under the above operating environment, this application provides Figure 3 The image processing method shown. Figure 3 Flowchart of an image processing method according to Embodiment 1 of the present invention. The image processing method provided by the embodiment of the present application includes:

[0044] Step S302, obtaining source posture and texture information according to the source image;

[0045] In the above step S302 of the present application, after the source image uploaded by the user is obtained, the source pose of the source image is obtained by pose estimation based on the source image; at the same time, feature extraction is performed based on the source image to obtain texture information.

[0046] Step S304, obtaining a first composite image according to the source image, the target posture and the source posture;

[0047] In the above step S304 of the present application, by reusing the source image in step S302, encoding is performed based on the source image and the source posture obtained in step S302 based on the source image, combined with the pre-obtained target posture, to obtain a content feature map, wherein the encoding process can be as follows: the input includes two branches, one is the source image I_s, and the other is the combination of the source posture P_s and the target posture P_t. The two branches are first pre-processed separately to obtain the initial coding, and then input into the PATN module together for information fusion and mutual guidance, and finally output the content feature map.

[0048] The coarse composite image is obtained by decoding based on the content feature map, that is, the first composite image in the embodiment of the present application, wherein the process of decoding based on the content feature map to obtain the coarse composite image can be as follows: after at least one layer of convolution and upsampling, the intermediate features learned by the convolution network are integrated, and the spatial domain size of the feature map is made consistent with the input image. Finally, the output is compressed to the range of [-1,1] through the range compression function (usually Tanh), and then converted into RGB values ​​of 0 to 255 for display to obtain a coarse composite image.

[0049] Step S306, obtaining a residual image according to the texture information and the first synthesized image;

[0050] In the above step S306 of the present application, in the process of obtaining the residual map, the embodiment of the present application provides two acquisition methods:

[0051] Method 1: Based on the texture information obtained in step S302 and the content feature map obtained in step S304, normalization processing is performed, and the features extracted by the convolutional neural network are restored into an image by reconstruction to obtain a residual map, wherein the features extracted by the convolutional neural network can be restored into an image by reconstruction, specifically by: restoring the features extracted by the convolutional neural network into an image by a method of convolution combined with upsampling, or by a deconvolution method. It should be noted that the reconstruction method in the embodiment of the present application is only illustrated by taking the above example as an example, and is subject to the method for implementing the image processing provided in the embodiment of the present application, and is not specifically limited.

[0052] Method 2: Perform normalization processing based on the texture information obtained in step S302 and the first synthesized image obtained in step S304 to obtain a residual image.

[0053] Among them, the residual image is an image that contains the contour features and surface texture details in the source image.

[0054] Step S308: acquiring a second synthesized image according to the first synthesized image and the residual image.

[0055] In the above step S308 of the present application, based on the first composite image obtained in step S304 and the residual image obtained in step S306, the residual image is superimposed on the coarse composite image to output a repaired fine image (Refined Image), wherein the fine image may be an image obtained by supplementing the contour features and surface texture details in the source image with the residual image on the coarse composite image. In the embodiment of the present application, the coarse composite image lacks the contour features and surface texture details in the residual image compared to the fine image, that is, the second composite image in the embodiment of the present application.

[0056] In summary, in combination with step S302 to step S308, Figure 4a-4c is a schematic diagram of obtaining a second synthesized image according to a source image in an image processing method according to Embodiment 1 of the present invention, such as Figure 4a-4c As shown, the image processing method provided in the embodiment of the present application can be applied to the field of e-commerce. For example, on the shopping website side, the seller uploads a product picture. If the product picture is a picture of a model wearing clothes for sale, but in order to generate an advertising text, the picture cannot meet the need to generate an advertising text, therefore, the clothes picture can be the source image in the embodiment of the present application. After the source picture is obtained, the target pose is obtained based on the requirements of the advertising text. Therefore, the source pose (Source Pose) of the source image is first obtained through a pose estimation algorithm based on the source image (Source Image); then, the pose transfer module (Pose Transfer Module) is used to encode based on the source image, the target pose (Target Pose) and the source pose to obtain a content feature map (Content Feature Map), and the content feature map is decoded by a decoder to obtain a coarse synthetic image, that is, the first synthetic image in the embodiment of the present application;

[0057] In addition, the image processing method provided in the embodiment of the present application further extracts features from the clothing image through a deep learning network to obtain texture information coding (that is, the texture information in the embodiment of the present application is embodied through texture information coding), and performs normalization (Adaptive Instance Normalization, AdaIN for short) based on the texture information coding (Texture Code) and the content feature map through a texture enhancement module (Texture Enhancing Module), and obtains a residual map (residual map) containing face and clothing texture details after deconvolution (deconv);

[0058] The coarse synthesized image is combined with the residual image, for example, the residual image is superimposed on the coarse synthesized image, and a repaired refined image (Refined Image) is output, that is, the second synthesized image in the embodiment of the present application is obtained.

[0059] That is, through the above method, on the shopping website side, after the seller uploads the picture of the clothes, more pictures of models wearing the clothes for sale are obtained through character posture migration, so as to realize the automatic synthesis of "seller show" video from static character images, realize the transformation of e-commerce promotion "from static to dynamic", enhance the promotion and display capabilities of small and medium-sized businesses, promote the updating and upgrading of e-commerce platforms, and enhance promotion methods and product value mining.

[0060] like Figure 4a-4c As shown in the figure, the target posture is represented by a skeleton graph, but in practical applications, the representation of posture features includes but is not limited to 2D / 3D skeletons, human body segmentation maps (global or local), parameterized / non-parametric three-dimensional human body models (including point clouds, polygonal meshes, voxels, etc.), etc.

[0061] Among them, Figure 4c As shown, based on Figure 4a and 4b The image processing flow is to generate a synthetic image by inputting the image. Figure 4c It can be seen that when the synthesized image is presented in a new posture, it can also reflect the facial features, hairstyle, and clothing texture of the model in the input image, thereby achieving the purpose of improving the lost / weakened detail information in the rough synthesized image, thereby achieving the technical effect of enhancing the realism of the synthesized image.

[0062] It should be noted that the image processing method provided in the embodiment of the present application is only illustrated by taking the above example as an example, and the image processing method provided in the embodiment of the present application shall prevail, and no specific limitation is made.

[0063] In an embodiment of the present invention, a method of multiplexing an input source image to extract texture information encoding and multiplexing a content feature map obtained based on the original image, the target posture and the source posture to provide guidance containing spatial information is adopted. By obtaining source posture and texture information based on the source image; obtaining a first synthetic image based on the source image, the target posture and the source posture; obtaining a residual map based on the texture information and the first synthetic image; and obtaining a second synthetic image based on the first synthetic image and the residual map, the purpose of improving the lost / weakened detail information in the rough synthetic image is achieved, thereby achieving the technical effect of improving the realism of the synthetic image, and further solving the technical problem that the prior art omits texture details when extracting features during character action migration, resulting in a lack of realism in the synthetic image.

[0064] Optionally, obtaining source posture and texture information according to the source image in step S302 includes:

[0065] Step S3021, obtaining a source pose of the source image by pose estimation according to the source image;

[0066] Step S3022: extract the source image by feature extraction to obtain the texture information of the source image.

[0067] Among them, the features of the source image are extracted through a deep learning network to obtain texture information, and the texture information is reflected in the form of texture information coding, that is, the source image is subjected to convolution, residual and average sampling processing to obtain the texture information coding of the source image.

[0068] Optionally, obtaining the first synthesized image according to the source image, the target posture and the source posture in step S304 includes: obtaining a content feature map according to the source image, the target posture and the source posture; and obtaining the first synthesized image according to the content feature map.

[0069] By multiplexing the source image, encoding is performed based on the source image and the source posture obtained according to the source image, combined with the pre-obtained target posture, to obtain a content feature map, wherein the encoding process can be as follows: the input includes two branches, one is the source image I_s, and the other is the combination of the source posture P_s and the target posture P_t. The two branches are first pre-processed separately to obtain the initial coding, and then input into the PATN module together for information fusion and mutual guidance, and finally output the content feature map.

[0070] The coarse composite image is obtained by decoding based on the content feature map, that is, the first composite image in the embodiment of the present application, wherein the process of decoding based on the content feature map to obtain the coarse composite image can be as follows: after at least one layer of convolution and upsampling, the intermediate features learned by the convolution network are integrated, and the spatial domain size of the feature map is made consistent with the input image. Finally, the output is compressed to the range of [-1,1] through the range compression function (usually Tanh), and then converted into RGB values ​​of 0 to 255 for display to obtain a coarse composite image.

[0071] Optionally, in step S306, obtaining a residual image according to the texture information and the first synthesized image includes:

[0072] Step S3061, performing normalization processing based on texture information coding and content feature map to obtain a residual map, wherein the normalization processing based on texture information coding and content feature map to obtain the residual map includes: obtaining the residual map through deep learning based on texture information coding, normalization with the content feature map, and reconstruction, wherein the residual map includes contour features and surface texture details in the source image.

[0073] Among them, based on the texture information coding, deep learning is used to adjust the dimension of the texture information coding to the size required by the network without affecting the information contained in the texture information coding; then the texture information coding after deep learning is combined with the content features Figure 1 Through normalization, the mean and variance of the deep features are controlled, thereby achieving the migration and conversion of the "style". Among them, the normalization processing in the embodiment of the present application can be obtained by processing an adaptive instance normalization layer (Adaptive Instance Normalization, referred to as AdaIN).

[0074] Finally, through deconvolution, the features extracted by the convolutional neural network are restored to an image, and the resolution is improved while maintaining the general structure of the image. The features of different layers are fused to obtain the residual map.

[0075] Further, optionally, the contour features include: at least one of human face, animal head, limb features or object appearance features; the surface texture details include: product surface texture details, wherein the product surface texture details include: at least one of clothing texture details, accessory texture details or utensil texture details.

[0076] Specifically, in the embodiment of the present application, still taking the source image of the model wearing the clothes for sale as an example, the residual image obtained after processing in step S3061 is an image containing the face and the texture details of the clothes, where the face is a kind of contour feature, and the texture feature of the clothes is a kind of surface texture detail.

[0077] If the source image is an image of pet supplies, such as an image of a dog wearing a pet raincoat, the residual image obtained in step S3061 may be an image including the dog's head features and / or torso features, as well as the surface texture of the raincoat.

[0078] Optionally, obtaining the second synthesized image according to the first synthesized image and the residual image in step S308 includes: step S3081, superimposing the first synthesized image and the residual image to obtain the second synthesized image.

[0079] Furthermore, in step S3081, superimposing the first composite image and the residual image to obtain the second composite image includes: filling in the positions corresponding to the contour features and surface texture details in the first composite image according to the contour features and surface texture details in the source image in the residual image to obtain the second composite image, wherein the second composite image has the contour features and surface texture details in the source image compared to the first composite image.

[0080] The key to the image processing method provided in the embodiment of the present application is to reuse the input source image to extract the texture information code z_t, and reuse the content feature map obtained in step S304 to provide guidance containing spatial information, instructing the texture enhancement module to generate a suitable texture at the correct position, thereby completing, that is, supplementing the lost texture details by generation, thereby improving the lost / weakened detail information in the rough synthesized image, and then significantly improving the visual quality of the synthesized image.

[0081] Among them, spatial domain information refers to generating the correct texture in the correct position. For example, the face area needs to correctly synthesize the eyes, nose, and mouth; the clothing area needs to generate appropriate patterns and fabrics, etc.

[0082] Example 2

[0083] According to another aspect of the embodiment of the present invention, a method for data processing is also provided. Figure 5 is a flow chart of a method for data processing according to Embodiment 2 of the present invention. Figure 5 As shown, the data processing method provided in the embodiment of the present application includes:

[0084] Step S502, receiving a source image uploaded by a user;

[0085] Step S504, obtaining source posture and texture information of the source image according to the source image;

[0086] Step S506, performing posture migration according to the source image, the target posture and the source posture to obtain a first synthetic image;

[0087] Step S508, performing feature enhancement processing according to the texture information to obtain a residual image, wherein the residual image includes: contour features and surface texture details in the source image;

[0088] Step S510, filling the first synthetic image with contour features and surface texture details in the residual image to generate a second synthetic image;

[0089] Step S512: determine the second synthesized image as a display image for display.

[0090] In summary, in combination with step S502 to step S512, the data processing method provided by the embodiment of the present application can be applied to the field of e-commerce. For example, on the shopping website side, the seller uploads a product picture. If the product picture is a picture of a model wearing clothes for sale, but in order to generate an advertising text, the picture cannot meet the need to generate an advertising text, therefore, the clothes picture can be the source image in the embodiment of the present application. After obtaining the source picture, the target pose is obtained based on the requirements of the advertising text. Therefore, according to the source image (Source Image), the source pose (SourcePose) of the source image is first obtained through a pose estimation algorithm; then, the pose transfer module (Pose Transfer Module) is used to encode based on the source image, the target pose (TargetPose) and the source pose to obtain a content feature map (Content Feature Map), and the content feature map is decoded by a decoder to obtain a rough synthetic image, that is, the first synthetic image in the embodiment of the present application;

[0091] In addition, the data processing method provided in the embodiment of the present application further extracts features from the clothing image through a deep learning network to obtain texture information coding, and performs normalization (Adaptive Instance Normalization, AdaIN for short) based on the texture information coding (Texture Code) and the content feature map obtained by the above process through a texture enhancement module (Texture Enhancing Module), and obtains a residual map (residual map) containing face and clothing texture details after deconvolution (deconv);

[0092] The coarse synthesized image is combined with the residual image, for example, the residual image is superimposed on the coarse synthesized image, and a repaired refined image (Refined Image) is output, that is, the second synthesized image in the embodiment of the present application is obtained.

[0093] That is, through the above method, on the shopping website side, after the seller uploads the picture of the clothes, more pictures of models wearing the clothes for sale are obtained through character posture migration, so as to realize the automatic synthesis of "seller show" video from static character images, realize the transformation of e-commerce promotion "from static to dynamic", enhance the promotion and display capabilities of small and medium-sized businesses, promote the update and upgrade of the company's e-commerce platform, and enhance the promotion methods and product value mining.

[0094] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0095] Through the description of the above implementation methods, those skilled in the art can clearly understand that the data processing method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0096] Example 3

[0097] According to another aspect of the embodiments of the present invention, a method for image processing is also provided. Figure 6 FIG. 1 is a flow chart of an image processing method according to Embodiment 3 of the present invention. Figure 6 As shown, the image processing method provided in the embodiment of the present application includes:

[0098] Step S602, receiving a source image uploaded by a user;

[0099] Step S604, generating a first synthetic image according to the source posture of the source image, and generating a residual image according to the texture information of the source image;

[0100] Step S606, obtaining a second synthesized image according to the first synthesized image and the residual image;

[0101] Step S608: Generate an image set or video data according to the second synthesized image.

[0102] Optionally, the image collection or video data is used for online fitting effect display or advertising page display.

[0103] In summary, in combination with step S602 to step S608, the image processing method provided in the embodiment of the present application can be applied to the field of e-commerce, that is, on the shopping website side, for example, on the backend server side of the online shopping platform, a product picture uploaded by the seller is received. If the product picture is a picture of a model wearing clothes for sale, but in order to generate an advertising text, the picture cannot meet the need to generate an advertising text, therefore, the clothes picture can be the source image in the embodiment of the present application. After obtaining the source picture, the target pose is obtained based on the requirements of the advertising text. Therefore, according to the source image (Source Image), the source pose (Source Pose) of the source image is first obtained through a pose estimation algorithm; then, the source image, the target pose (Target Pose) and the source pose are encoded through a pose transfer module (Pose Transfer Module) to obtain a content feature map (Content Feature Map), and the content feature map is decoded through a decoder to obtain a rough synthetic image, that is, the first synthetic image in the embodiment of the present application;

[0104] In addition, the image processing method provided in the embodiment of the present application further extracts features from the clothing image through a deep learning network to obtain texture information coding (that is, the texture information in the embodiment of the present application is embodied through texture information coding), and performs normalization (Adaptive Instance Normalization, AdaIN for short) based on the texture information coding (Texture Code) and the content feature map through a texture enhancement module (Texture Enhancing Module), and obtains a residual map (residual map) containing face and clothing texture details after deconvolution (deconv);

[0105] The coarse synthesized image is combined with the residual image, for example, the residual image is superimposed on the coarse synthesized image, and a repaired refined image (Refined Image) is output, that is, the second synthesized image in the embodiment of the present application is obtained.

[0106] That is, through the above method, on the shopping website side, after the seller uploads the picture of the clothes, more pictures of models wearing the clothes for sale are obtained through character posture migration, so as to realize the automatic synthesis of a "seller show" video (or a promotional page composed of multiple synthesized pictures) from static character images, realize the transformation of e-commerce promotion "from static to dynamic", enhance the promotion and display capabilities of small and medium-sized businesses, promote the updating and upgrading of e-commerce platforms, and enhance promotional methods and product value mining.

[0107] In addition, on the buyer's side, when a user is browsing an online e-commerce website and is interested in a piece of clothing, accessories, or appliances for sale, the user can turn on the camera and obtain a user image through the terminal, where the user image includes: the user's facial features (facial features, skin texture, hairstyle, and whether the face is made up), the user's torso features (the current user's posture, for example, standing or sitting; and the clothes the user is currently wearing); based on the user image obtained above, the server side synthesizes an image of the user wearing or taking the clothes, accessories, or appliances for sale, and generates multiple images or videos based on the image, and feeds the multiple images or videos back to the client for the user to browse, so that the user can fully understand and experience the wearing effect or use effect of the clothes, accessories, and appliances for sale.

[0108] Among them, if the terminal used by the user is an AR or VR device, the generated image or video will be able to adapt to the AR or VR device for display.

[0109] It should be noted that the image processing method provided in the embodiment of the present application is only illustrated by taking the above example as an example, and the image processing method provided in the embodiment of the present application shall prevail, and no specific limitation is made.

[0110] In an embodiment of the present invention, a method of multiplexing an input source image to extract texture information encoding, and multiplexing a content feature map obtained based on the original image, the target posture and the source posture to provide guidance containing spatial information is adopted. The source image uploaded by the user is received; a first synthetic image and a residual image are generated based on the source posture and texture information of the source image; a second synthetic image is obtained based on the first synthetic image and the residual image; and an image set or video data is generated based on the second synthetic image. The purpose of improving the lost / weakened detail information in the rough synthetic image is achieved, thereby achieving the technical effect of improving the realism of the synthetic image, and further solving the technical problem that the prior art omits texture details when extracting features during character action migration, resulting in a lack of realism in the synthetic image.

[0111] Example 4

[0112] According to another aspect of the embodiments of the present invention, a method for acquiring an image is provided. Figure 7 FIG. 4 is a flow chart of a method for acquiring an image according to Embodiment 4 of the present invention. Figure 7 As shown, the image acquisition method provided in the embodiment of the present application includes:

[0113] Step S702, uploading the source image;

[0114] Step S704: receiving an image set or video data returned based on the source image.

[0115] Among them, the image set or video data generates a first synthetic image and a residual image based on the source posture and texture information of the source image; obtains a second synthetic image based on the first synthetic image and the residual image; and obtains based on the second synthetic image.

[0116] Specifically, in combination with steps S702 to S704, on the client side used by the user, the user opens the smart terminal currently used, where the smart terminal may include: smart phones, laptops, desktop computers, wearable smart devices (for example, smart watches, AR, VR devices), and other terminals with cameras or data transmission interfaces.

[0117] Taking a smartphone as an example, a user opens the camera of the smartphone, and the client APP installed in the smartphone obtains the user's image, where the user image includes: the user's facial features (facial features, skin texture, hairstyle, whether the face is made up), the user's torso features (the current user's posture, for example, standing or sitting; and the clothes the user is currently wearing);

[0118] If the user is interested in a piece of clothing, accessories, or appliances for sale, the client APP will prompt the user to capture the user's current behavior posture through the camera, and upload the image with the behavior posture as the source image. The server will synthesize an image of the user wearing or taking the clothes, accessories, or appliances for sale based on the source image, and generate multiple images or videos based on the image. The multiple images or videos will be fed back to the client for the user to browse, so that the user can fully understand and experience the wearing effect or use effect of the clothes, accessories, and appliances for sale.

[0119] Among them, if the terminal used by the user is an AR or VR device, the generated image or video will be able to adapt to the AR or VR device for display.

[0120] It should be noted that the image processing method provided in the embodiment of the present application is only illustrated by taking the above example as an example, and the image processing method provided in the embodiment of the present application shall prevail, and no specific limitation is made.

[0121] In the embodiment of the present invention, the source image input is reused to extract texture information encoding, and the content feature map obtained based on the original image, the target posture and the source posture is reused to provide guidance containing spatial information. By uploading the source image and receiving the image set or video data returned based on the source image, the purpose of improving the lost / weakened detail information in the rough synthetic image is achieved, thereby achieving the technical effect of improving the realism of the synthetic image, and further solving the technical problem that the synthetic image lacks realism due to the omission of texture details when extracting features when performing character action migration in the prior art. Embodiment 5

[0122] According to an embodiment of the present invention, there is also provided a device for implementing the image processing method in the above embodiment 1. Figure 8 is a schematic diagram of an image processing apparatus according to a fifth embodiment of the present invention. Figure 8 As shown, the device includes: a first acquisition module 82, used to acquire source posture and texture information according to a source image; a first synthesis module 84, used to acquire a first synthetic image according to the source image, the target posture and the source posture; a second acquisition module 86, used to acquire a residual image according to the texture information and the first synthetic image; a second synthesis module 88, used to acquire a second synthetic image according to the first synthetic image and the residual image.

[0123] Example 6

[0124] According to an embodiment of the present invention, there is also provided a device for implementing the data processing method in the above embodiment 2. Fig. 9 is a schematic diagram of a data processing device according to a sixth embodiment of the present invention. Fig. 9 As shown, it includes: a receiving module 91, which is used to receive a source image uploaded by a user; an acquisition module 92, which is used to acquire the source posture and texture information of the source image based on the source image; a first image processing module 93, which is used to perform posture migration based on the source image, the target posture and the source posture, and acquire a first synthetic image; a second image processing module 94, which is used to perform feature enhancement processing based on the texture information, and acquire a residual image, wherein the residual image includes: contour features and surface texture details in the source image; a synthesis module 95, which is used to fill the contour features and surface texture details in the residual image in the first synthetic image, and generate a second synthetic image; a display module 96, which is used to determine the second synthetic image as a display image for display.

[0125] Example 7

[0126] According to another aspect of another embodiment of the present invention, there is also provided an image processing device, which includes: a receiving module for receiving a source image uploaded by a user; a first generating module for generating a first synthetic image based on a source posture of the source image, and generating a residual image based on texture information of the source image; an acquiring module for acquiring a second synthetic image based on the first synthetic image and the residual image; and a second generating module for generating an image set or video data based on the second synthetic image.

[0127] Example 8

[0128] According to another aspect of another embodiment of the present invention, there is also provided an image acquisition device, which includes: an upload module for uploading a source image; a receiving module for receiving an image set or video data returned based on the source image, wherein the image set or video data generates a first composite image and a residual image based on the source posture and texture information of the source image; a second composite image is acquired based on the first composite image and the residual image; and is obtained based on the second composite image.

[0129] Example 9

[0130] According to one aspect of another embodiment of the present invention, a storage medium is also provided, wherein the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the image and data processing method in the above-mentioned embodiment 1 or 2.

[0131] Example 10

[0132] According to one aspect of yet another embodiment of the present invention, a processor is provided, wherein the processor is used to run a program, wherein when the program is run, the method for image and data processing in the above-mentioned embodiment 1 or 2 is executed.

[0133] Embodiment 11

[0134] The embodiment of the present invention further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image processing method provided in the first embodiment.

[0135] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0136] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining source posture and texture information based on a source image; obtaining a first synthetic image based on the source image, the target posture and the source posture; obtaining a residual map based on the texture information and the first synthetic image; obtaining a second synthetic image based on the first synthetic image and the residual map.

[0137] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining source posture and texture information according to the source image includes: obtaining the source posture of the source image through posture estimation according to the source image; obtaining the texture information of the source image through feature extraction of the source image.

[0138] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining a first composite image based on a source image, a target posture and a source posture includes: obtaining a content feature map based on the source image, the target posture and the source posture; obtaining a first composite image based on the content feature map.

[0139] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining a residual map based on the texture information and the first synthetic image includes: performing normalization processing based on the texture information encoding and the content feature map to obtain a residual map; wherein, performing normalization processing based on the texture information encoding and the content feature map to obtain a residual map includes: obtaining a residual map through deep learning based on the texture information, normalizing with the content feature map, and reconstructing, wherein the residual map includes contour features and surface texture details in the source image.

[0140] Further, optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: contour features include: at least one of human face, animal head, limb features or object appearance features; surface texture details include: commodity surface texture details, wherein the commodity surface texture details include: at least one of clothing texture details, jewelry texture details or utensil texture details.

[0141] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: acquiring the second composite image according to the first composite image and the residual map includes: obtaining the second composite image by superimposing the first composite image and the residual map.

[0142] Further, optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: superimposing the first composite image and the residual map to obtain a second composite image includes: filling in positions corresponding to the contour features and surface texture details in the first composite image based on the contour features and surface texture details in the source image in the residual map to obtain the second composite image, wherein the second composite image has the contour features and surface texture details in the source image compared to the first composite image.

[0143] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0144] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0148] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0149] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for image processing, wherein: include: Obtain source pose and texture information according to the source image; Acquire a content feature map according to the source image, the target posture and the source posture; Acquire a first composite image according to the source image, the target posture and the source posture; Acquire a residual image according to the texture information and the content feature map, wherein the residual image is obtained by restoring the features extracted by the convolutional neural network into an image; A second synthesized image is acquired according to the first synthesized image and the residual image.

2. The method according to claim 1, wherein: Obtaining source pose and texture information based on the source image includes: Obtaining a source pose of the source image by pose estimation according to the source image; The texture information of the source image is obtained by extracting features from the source image.

3. The method according to claim 1, wherein: Acquiring a first composite image according to the source image, the target posture and the source posture comprises: Acquire a content feature map according to the source image, the target posture and the source posture; A first synthesized image is acquired according to the content feature map.

4. The method according to claim 3, wherein: Acquiring a residual map according to the texture information and the first synthesized image includes: Performing normalization processing according to the texture information coding and the content feature map to obtain a residual map; wherein performing normalization processing according to the texture information coding and the content feature map to obtain a residual map includes: The residual map is obtained through deep learning based on the texture information, normalization with the content feature map, and reconstruction, wherein the residual map includes contour features and surface texture details in the source image.

5. The method according to claim 4, wherein: The contour features include: at least one of human face, animal head, limb features or object appearance features; the surface texture details include: commodity surface texture details, wherein the commodity surface texture details include: at least one of clothing texture details, jewelry texture details or utensil texture details.

6. The method according to claim 4, wherein: Acquiring a second synthesized image according to the first synthesized image and the residual image includes: The first synthesized image and the residual image are superimposed to obtain a second synthesized image.

7. The method according to claim 6, wherein: Obtaining a second synthesized image by superimposing the first synthesized image and the residual image includes: Based on the contour features and surface texture details in the source image in the residual image, the positions corresponding to the contour features and the surface texture details in the first synthetic image are filled to obtain the second synthetic image, wherein the second synthetic image has the contour features and the surface texture details in the source image compared to the first synthetic image.

8. A method of data processing, wherein: include: Receive source images uploaded by users; Acquiring source posture and texture information of the source image according to the source image; Performing posture migration according to the source image, the target posture and the source posture to obtain a first synthetic image; Performing feature enhancement processing according to the texture information to obtain a residual image, wherein the residual image includes: contour features and surface texture details in the source image; Filling the contour features and the surface texture details in the residual image in the first synthetic image to generate a second synthetic image; The second composite image is determined as a display image for display.

9. A method for image processing, wherein: include: Receive source images uploaded by users; generating a first synthetic image according to a source pose of the source image, and generating a residual image according to texture information of the source image; The step of generating a first synthetic image according to the source posture of the source image includes: Encoding the source image, the target posture, and the source posture to obtain a content feature map; Decoding the content feature map by a decoder to obtain the first synthesized image; Generating a residual image according to texture information of the source image, comprising: Performing normalization and deconvolution operations on the texture information and the content feature map to obtain the residual image; Acquire a second synthesized image according to the first synthesized image and the residual image; A set of images or video data is generated based on the second composite image.

10. The method according to claim 9, wherein: The image collection or video data is applied to online fitting effect display or advertisement page display.

11. A method for acquiring an image, wherein: include: Upload the source image; Receive an image set or video data returned based on the source image, wherein the image set or the video data is generated based on a first synthetic image and a residual image, the first synthetic image is generated by decoding a content feature map through a decoder, the content feature map is obtained by encoding the source image, the target posture, and the source posture, and the residual image is generated based on texture information of the source image by normalizing and deconvolving the texture information and the content feature map.

12. An image processing device, wherein: include: A first acquisition module, used for acquiring source posture and texture information according to a source image; The first acquisition module is also used to acquire a content feature map according to the source image, the target posture and the source posture; A first synthesis module, used for obtaining a first synthesized image according to the source image, the target posture and the source posture; A second acquisition module is used to acquire a residual image according to the texture information and the content feature map, wherein the residual image is obtained by restoring the features extracted by the convolutional neural network into an image; The second synthesis module is used to obtain a second synthesized image according to the first synthesized image and the residual image.

13. A data processing device, wherein: include: A receiving module, used for receiving a source image uploaded by a user; An acquisition module, used for acquiring source posture and texture information of the source image according to the source image; A first image processing module, used for performing posture migration according to the source image, the target posture and the source posture to obtain a first synthetic image; A second image processing module is used to perform feature enhancement processing according to the texture information to obtain a residual image, wherein the residual image includes: contour features and surface texture details in the source image; A synthesis module, used for filling the contour features and the surface texture details in the residual image in the first synthetic image to generate a second synthetic image; A display module is used to determine the second composite image as a display image for display.

14. An image processing device, wherein: include: A receiving module, used for receiving a source image uploaded by a user; A first generating module, configured to generate a first synthesized image according to a source posture of the source image, and to generate a residual image according to texture information of the source image; The first generation module is further used to encode the source image, the target posture, and the source posture to obtain a content feature map; decode the content feature map through a decoder to obtain the first synthetic image; and perform normalization and deconvolution operations on the texture information and the content feature map to obtain the residual image; An acquisition module, configured to acquire a second composite image based on the first composite image and the residual image; The second generating module is used to generate an image set or video data according to the second synthesized image.

15. An image acquisition device, wherein: include: Upload module, used to upload source images; A receiving module is used to receive an image set or video data returned based on the source image, wherein the image set or the video data is generated based on a first synthetic image and a residual image, the first synthetic image is generated by decoding a content feature map through a decoder, the content feature map is obtained by encoding the source image, the target posture, and the source posture, and the residual image is generated based on the texture information of the source image by normalizing and deconvolving the texture information and the content feature map.

16. A storage medium, wherein: The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the method for image and data processing described in any one of claims 1 to 11.

17. A processor, wherein: The processor is used to run a program, wherein the program, when running, executes the method for image and data processing described in any one of claims 1 to 11.

Images