Image processing method and device, electronic equipment and computer readable storage medium

Abstract

Embodiments of the present application disclose an image processing method and device, electronic equipment and computer readable storage medium; the embodiments of the present application obtain the original image of original object and the driving image of driving object after obtaining the original image of original object and the driving image of driving object, the object feature of original object is extracted in original image, and the posture feature of driving object is extracted in driving image, the object feature and posture feature are fused, three-dimensional image feature is obtained, and three-dimensional object image is constructed based on three-dimensional image feature, the spatial feature under each preset resolution is extracted in original image and three-dimensional object image, the original spatial feature of original image and the three-dimensional spatial feature of three-dimensional object image are obtained, the original spatial feature, three-dimensional spatial feature and preset basic style feature are fused to obtain target object image;The scheme can improve the accuracy of image processing, and the embodiments of the present application can be applied to cloud technology, artificial intelligence, intelligent transportation, auxiliary driving and various scenes.

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and in particular to an image processing method, device, electronic device, and computer-readable storage medium. Background Art

[0002] In recent years, with the development of internet technology, image processing has become increasingly diverse. For example, the pose of an object in an original image can be adjusted to the pose of the object in the driving image through a driving image. Existing image processing methods represent poses through keypoint trajectories or local affine transformations of keypoints in the image, thereby achieving pose transfer.

[0003] During the research and practice of the existing technology, the inventors of the present invention found that for existing image processing methods, the key point trajectory cannot accurately capture the local posture details of the object in the object image, and is subject to the influence of the object image resolution, making it impossible to restore the high-resolution image, thereby greatly reducing the accuracy of image processing. Summary of the Invention

[0004] Embodiments of the present invention provide an image processing method, apparatus, electronic device, and computer-readable storage medium, which can improve the accuracy of image processing.

[0005] An image processing method, comprising:

[0006] Acquire an original image of the original object and a driven image of the driven object, wherein the driven image is used as a template image for adjusting the body posture of the original object;

[0007] extracting object features of the original object from the original image, and extracting posture features of the driven object from the driven image;

[0008] fusing the object features and the posture features to obtain three-dimensional image features, and constructing a three-dimensional object image based on the three-dimensional image features;

[0009] Extracting spatial features at each preset resolution from the original image and the three-dimensional object image to obtain original spatial features of the original image and three-dimensional spatial features of the three-dimensional object image;

[0010] The original spatial features, the three-dimensional spatial features and the preset basic style features are fused to obtain a target object image at a target resolution, wherein the target object image is an object image in which the body posture of the original object is replaced with the body posture of the driven object.

[0011] Optionally, this solution may also provide an image processing method, including:

[0012] Acquire an original facial image and a facial driving image, wherein the facial driving image is a template image for adjusting a facial posture in the original facial image;

[0013] Extracting facial image features from the original facial image and extracting facial posture features from the facial drive image;

[0014] fusing the facial image features and facial posture features to obtain three-dimensional facial features, and constructing a three-dimensional facial image based on the three-dimensional facial image;

[0015] Extracting spatial features at each preset resolution from the original facial image and the facial drive image to obtain original spatial features of the original facial image and three-dimensional spatial features of the three-dimensional facial image;

[0016] The original spatial features, the three-dimensional spatial features, and the preset basic style features are fused to obtain a target facial image at a target resolution, wherein the target facial image is an image in which the facial posture in the original facial image is replaced with the facial posture in the facial driving image.

[0017] Accordingly, an embodiment of the present invention provides an image processing device, including:

[0018] an acquisition unit, configured to acquire an original image of the original object and a driving image of the driving object, wherein the driving image is used as a template image for adjusting the body posture of the original object;

[0019] a first extraction unit, configured to extract object features of the original object from the original image, and extract posture features of the driven object from the driven image;

[0020] a construction unit, configured to fuse the object features and the posture features to obtain three-dimensional image features, and construct a three-dimensional object image based on the three-dimensional image features;

[0021] a second extraction unit, configured to extract spatial features at each preset resolution from the original image and the three-dimensional object image, to obtain original spatial features of the original image and three-dimensional spatial features of the three-dimensional object image;

[0022] A fusion unit is used to fuse the original spatial features, the three-dimensional spatial features and the preset basic style features to obtain a target object image at a target resolution, wherein the target object image is an object image in which the body posture of the original object is replaced with the body posture of the driven object.

[0023] Optionally, in some embodiments, the fusion unit can be specifically used to generate a basic object image at an initial resolution based on the preset basic style features; use a trained image generation model to map the three-dimensional image features into latent features, and adjust the preset basic style features based on the latent features; and according to the basic object image, fuse the adjusted style features, original spatial features and three-dimensional spatial features to obtain a target object image at a target resolution.

[0024] Optionally, in some embodiments, the fusion unit can be specifically used to filter out the target original spatial features from the original spatial features based on the preset resolution, and filter out the target three-dimensional spatial features from the three-dimensional spatial features; fuse the adjusted style features, the target original spatial features and the target three-dimensional spatial features to obtain the fused style features at the current resolution; and generate the target object image at the target resolution based on the fused style features and the basic object image.

[0025] Optionally, in some embodiments, the fusion unit can be specifically used to generate a current object image based on the fused style features, and fuse the current object image with the basic object image to obtain a fused object image at the current resolution; use the fused style features as the preset basic style features, and use the fused object image as the basic object image; return to execute the step of adjusting the preset basic style features based on the latent features until the current resolution is the target resolution, and obtain the target object image.

[0026] Optionally, in some embodiments, the fusion unit can be specifically used to adjust the size of the basic style feature to obtain the initial style feature; modulate the latent feature to obtain the convolution weight corresponding to the initial style feature; and adjust the initial style feature based on the convolution weight to obtain the adjusted style feature.

[0027] Optionally, in some embodiments, the fusion unit can be specifically used to filter out a target style convolutional network corresponding to the resolution of the base object image in the style convolutional network of the trained image generation model; based on the convolution weight, the target style convolutional network is used to perform convolution processing on the initial style feature to obtain an adjusted style feature.

[0028] Optionally, in some embodiments, the fusion unit can be specifically used to generate a basic optical flow field at the initial resolution based on the basic style features; according to the basic optical flow field, the adjusted style features, the original spatial features, and the three-dimensional spatial features are fused to obtain a target optical flow field at the target resolution.

[0029] Optionally, in some embodiments, the image processing device may further include a training unit, which may be specifically used to obtain video samples and select any two video frames from the video samples as original image samples and driving image samples respectively; use a preset image generation model to perform object driving on the original image samples and the driving image samples to obtain a predicted object image; based on the predicted object image and the driving image samples, converge the preset image generation model to obtain a trained image generation model.

[0030] Optionally, in some embodiments, the training unit can be specifically used to detect the predicted object image and the driving image samples respectively, and determine the adversarial loss information of the preset image generation model based on the detection results; determine the reconstruction loss information of the preset image generation model based on the predicted object image and the driving image samples; fuse the adversarial loss information and the reconstruction loss information, and converge the preset image generation model based on the fused loss information to obtain a trained image generation model.

[0031] Optionally, in some embodiments, the training unit can be specifically used to calculate the image similarity between the predicted object image and the driving image sample to obtain the perceptual loss information of the preset image generation model; compare the predicted object image and the driving image sample to obtain the image loss information of the preset image generation model; and fuse the perceptual loss information and the image loss information to obtain the reconstruction loss information of the preset image generation model.

[0032] Optionally, in some embodiments, the construction unit can be specifically used to extract image features from the original image and the driving image respectively to obtain the original image features of the original image and the driving image features of the driving image; filter out the object features of the original object from the original image features; and filter out the posture features of the driving object from the driving image features.

[0033] Optionally, in some embodiments, the first extraction unit can be specifically used to extract image features from the original image and the driving image respectively to obtain the original image features of the original image and the driving image features of the driving image; filter out the object features of the original object from the original image features; and filter out the posture features of the driving object from the driving image features.

[0034] In addition, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores an application program, and the processor is configured to run the application program in the memory to implement the image processing method provided by the embodiment of the present invention.

[0035] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores a plurality of instructions suitable for loading by a processor to execute the steps of any image processing method provided by the embodiment of the present invention.

[0036] After acquiring the original image of the original object and the driving image of the driving object, the embodiment of the present invention extracts the object features of the original object from the original image and extracts the posture features of the driving object from the driving image. Then, the object features and the posture features are fused to obtain three-dimensional image features, and a three-dimensional object image is constructed based on the three-dimensional image features. Then, spatial features at each preset resolution are extracted from the original image and the three-dimensional object image to obtain the original spatial features of the original image and the three-dimensional spatial features of the three-dimensional object image. Then, the original spatial features, the three-dimensional spatial features and the preset basic style features are fused to obtain a target object image at a target resolution. Since the scheme can construct a three-dimensional object image based on the three-dimensional object features extracted from the original image and the driving image, the layout posture details of the driving object can be accurately captured, and the target object image at the target resolution is generated through the spatial features of the driving image and the three-dimensional reconstructed image at different resolutions, thereby ensuring the resolution of the generated target object image, thereby improving the accuracy of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 is a schematic diagram of a scenario of an image processing method provided by an embodiment of the present invention;

[0039] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of body posture adjustment provided by an embodiment of the present invention;

[0041] Figure 4 is a schematic diagram of 3D reconstruction provided by an embodiment of the present invention;

[0042] Figure 5 Schematic diagram of the network structure of the decoding subnetwork provided by an embodiment of the present invention;

[0043] Figure 6Schematic diagram of the overall training process of the preset image generation model provided by an embodiment of the present invention;

[0044] Figure 7 is a schematic diagram of the use process of the trained image model provided by an embodiment of the present invention;

[0045] Figure 8 is another flowchart of the image processing method provided by an embodiment of the present invention;

[0046] Figure 9 is another flowchart of the image processing method provided by an embodiment of the present invention;

[0047] Figure 10 is a schematic diagram of facial posture adjustment provided by an embodiment of the present invention;

[0048] Figure 11 is a schematic diagram of facial 3D reconstruction provided by an embodiment of the present invention;

[0049] Figure 12 This is a schematic diagram of a process for training a preset image generation model using facial images provided by an embodiment of the present invention;

[0050] Figure 13 1 is a schematic diagram of a process of performing facial driving on a face image by a trained image generation model provided by an embodiment of the present invention;

[0051] Figure 14 is a structural diagram of an image processing device provided by an embodiment of the present invention;

[0052] Figure 15 is another structural diagram of the image processing device provided by an embodiment of the present invention;

[0053] Figure 16 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0055] The embodiments of the present invention provide an image processing method, apparatus, electronic device, and computer-readable storage medium. The image processing apparatus can be integrated into an electronic device, which can be a server, a terminal, or other device.

[0056] Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. Terminals include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0057] For example, see Figure 1 Taking the image processing device integrated in an electronic device as an example, after obtaining the original image of the original object and the driving image of the driving object, the electronic device extracts the object features of the original object in the original image and extracts the posture features of the driving object in the driving image. Then, the object features and the posture features are fused to obtain three-dimensional image features, and based on the three-dimensional image features, a three-dimensional object image is constructed. Then, spatial features at each preset resolution are extracted from the original image and the three-dimensional object image to obtain the original spatial features of the original image and the three-dimensional spatial features of the three-dimensional object image. Then, the original spatial features, the three-dimensional spatial features and the preset basic style features are fused to obtain the target object image at the target resolution, thereby improving the accuracy of image processing.

[0058] The image processing method provided in the embodiments of the present application relates to computer vision in the field of artificial intelligence. The embodiments of the present application can extract features from the original image and the driven image, and drive the original image based on the driven image to obtain an image of the target object at a target resolution.

[0059] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. AI software technologies primarily include computer vision and machine learning / deep learning.

[0060] Computer vision (CV) is the science of making machines "see." Specifically, it refers to the use of computers to identify and measure objects, replacing the human eye. Further image processing is performed to make images more suitable for human observation or for transmission to instrumentation. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technology generally includes techniques such as image processing and image recognition.

[0061] It can be understood that in the specific implementation of this application, related data such as the original object image and the driven object image of the object are involved. When the following embodiments of this application are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0062] It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0063] This embodiment will be described from the perspective of an image processing device, which can be specifically integrated into an electronic device, which can be a server or a terminal; wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC), a wearable device, a virtual reality device or other smart device that can perform image processing.

[0064] An image processing method, comprising:

[0065] An original image of an original object and a driving image of a driving object are obtained, wherein the driving image is used as a template image for adjusting the body posture of the original object, object features of the original object are extracted from the original image, and posture features of the driving object are extracted from the driving image, the object features and the posture features are fused to obtain three-dimensional image features, and a three-dimensional object image is constructed based on the three-dimensional image features, spatial features at each preset resolution are extracted from the original image and the three-dimensional object image, original spatial features of the original image and three-dimensional spatial features of the three-dimensional object image are obtained, the original spatial features, the three-dimensional spatial features and the preset basic style features are fused to obtain a target object image at a target resolution, wherein the target object image is an object image in which the body posture of the original object is replaced with the body posture of the driving object.

[0066] like Figure 2 As shown, the specific process of the image processing method is as follows:

[0067] 101. Acquire an original image of an original object and a driven image of a driven object.

[0068] The original object may be an object whose body posture is to be adjusted, and the original image may be an image containing the original object. The driving image is a template image used to adjust the body posture of the original object. Adjusting the body posture of the original object can be understood as replacing the body posture of the original object with the body posture of the driving object, thereby obtaining the target object image. The type of object here can be of various types, such as a human body, a facial object, or an animal, etc. Taking the human body as an example, the process of adjusting the body posture can be as follows: Figure 3 shown.

[0069] There are many ways to obtain the original image of the original object and the driven image of the driven object, which can be specifically as follows:

[0070] For example, the original image of the original object and the driven image of the driven object uploaded by the terminal can be directly obtained, or any two video frames containing the object can be extracted from the video, one of which can be used as the original image, and the object in the original image can be used as the original object, and the remaining frame can be used as the driven image, and the object in the driven image can be used as the driven object, or the object image containing the object can be filtered out from the image data, and any two of the object images can be filtered out as the original image and the driven image, and so on.

[0071] 102. Extract object features of the original object from the original image, and extract posture features of the driven object from the driven image.

[0072] Among them, object features can be understood as feature information other than posture that indicates the original object, for example, it may include identity, lighting, texture, etc. Posture features can be understood as feature information that indicates the posture of the driving object. The type of driving object is different, and the posture features of the driving object are also different. For example, when the driving object is a human body, the posture feature can be the body posture (pose) of the human body. When the driving object is a face, the posture feature can be facial posture (pose), gaze, and expression, etc.

[0073] There are many ways to extract the object features of the original object from the original image and the posture features of the driven object from the driven image, which can be specifically as follows:

[0074] For example, image features can be extracted from the original image and the driving image respectively to obtain the original image features of the original image and the driving image features of the driving image, and the object features of the original object can be filtered out from the original image features, and the posture features of the driving object can be filtered out from the driving image features.

[0075] Among them, there are many ways to perform image feature extraction on the original image and the driving image. For example, the image feature extraction network of the trained image generation model can be used to extract 3D coefficients (coeff) from the original image and the driving image respectively, and the 3D coefficients extracted from the original image are used as the original image features, and the 3D coefficients extracted from the driving image are used as the driving image features.

[0076] Among them, the 3D coefficient can be understood as the three-dimensional parameters of the object in the image. Different objects in the image have different 3D coefficients. When the object in the image is a human body, the 3D coefficient can be the face identity, lighting, texture, and pose. When the object in the image is a facial object, the 3D coefficient can be the face identity, lighting, texture, expression, pose, and gaze. The network structure of the image feature extraction network for extracting the 3D coefficient can be various, for example, it can be ResNet50 or other network structures. Taking the network structure of the image feature extraction network as ResNet50 as an example, the process of extracting the 3D coefficient can be shown as formula (1):

[0077]

[0078] Among them, coeff is the 3D coefficient, I lqIt can be the original image or the driven image.

[0079] After extracting the original image features and the driving image features, the object features of the original object can be directly filtered out from the original image features. For example, if the original object is a human body, parameters such as identity, light and shadow, and texture can be directly extracted from the original image features to obtain the object features of the original object. The posture features of the driving object can also be filtered out from the driving image features. For example, if the driving object is a human body, parameters such as posture can be directly filtered out from the driving image features to obtain the posture features of the driving object.

[0080] 103. The object features and the posture features are fused to obtain three-dimensional image features, and a three-dimensional object image is constructed based on the three-dimensional image features.

[0081] Among them, the three-dimensional image features can be understood as the image features for reconstructing the three-dimensional object.

[0082] The three-dimensional object image can be an object image that is projected onto a two-dimensional plane by rendering the three-dimensional image features into a three-dimensional object model, which can also be understood as 3D reconstruction. The reconstruction process can be as follows: Figure 4 shown.

[0083] There are many ways to fuse object features and posture features, which can be as follows:

[0084] For example, the object features and posture features can be directly spliced ​​to obtain three-dimensional image features, or the weighting parameters of the object features and posture features can be obtained, and based on the weighting parameters, the object features and posture features can be weighted respectively to obtain weighted object features and weighted posture features, and the weighted object features and weighted posture features can be fused to obtain three-dimensional image features.

[0085] After fusing the object features and posture features, a three-dimensional object image can be constructed based on the three-dimensional image features obtained after the fusion. There are many ways to construct a three-dimensional object image. For example, the three-dimensional image features can be converted into geometric features and texture features of the three-dimensional object. Based on the geometric features and texture features, a three-dimensional object model of the three-dimensional object can be constructed. The three-dimensional object model can be projected onto a two-dimensional plane to obtain a three-dimensional object image of the three-dimensional object.

[0086] Among them, the geometric features can be understood as the coordinate information of the key points of the 3D mesh structure of the three-dimensional object, and the texture features can be understood as the features indicating the texture information of the three-dimensional object. There are many ways to convert the three-dimensional image features into the geometric features and texture features of the three-dimensional object. For example, the position information of at least one key point can be extracted from the three-dimensional image features, the position information of the key point can be converted into the geometric features, the texture information of the original object can be extracted from the three-dimensional image features, and the texture features can be extracted from the texture information. The specific conversion process can be shown as formula (2):

[0087]

[0088] Among them, S is the geometric feature, T is the texture feature, and coeff is the three-dimensional image feature.

[0089] After converting the geometric features and texture features, a three-dimensional object model can be constructed, and the three-dimensional object model can be projected onto a two-dimensional plane to obtain a three-dimensional object image. There are many ways to obtain a three-dimensional object image. For example, the three-dimensional model parameters of the three-dimensional object can be determined based on the geometric features and texture features. Based on the three-dimensional model parameters, a three-dimensional object model of the three-dimensional object can be constructed, and the three-dimensional object model can be projected onto a two-dimensional plane to obtain a three-dimensional object image. Specifically, it can be shown as formula (3):

[0090]

[0091] Among them, I 3d is a three-dimensional object image, S is the geometric feature, and T is the texture feature.

[0092] Among them, constructing a three-dimensional object image can be understood as performing 3D reconstruction of the three-dimensional object based on three-dimensional image features. There are many ways to reconstruct the three-dimensional object, for example, the SMPL reconstruction method, the 3DMMM reconstruction method or other 3D reconstruction methods can be used.

[0093] 104. Extract spatial features at each preset resolution from the original image and the three-dimensional object image to obtain original spatial features of the original image and three-dimensional spatial features of the three-dimensional object image.

[0094] The spatial feature can be understood as the spatial information representing each preset resolution in the original image or the three-dimensional object image. Therefore, the original spatial feature of the original image and the three-dimensional spatial feature of the three-dimensional object can both be multi-layer spatial features.

[0095] There are many ways to extract spatial features at each preset resolution from the original image and the three-dimensional object, which can be specifically as follows:

[0096] For example, the encoding network (Enc Block) of the trained image generation model can be used to spatially encode the original image and the three-dimensional object image at each preset resolution, thereby obtaining the original spatial features of the original image and the three-dimensional spatial features of the three-dimensional object image at each resolution.

[0097] Among them, the encoding network (Enc Block) can include multiple sub-encoding networks, each sub-encoding network corresponds to a preset resolution, and the sub-encodings can be arranged in order from small to large according to the size of the resolution, thereby obtaining an encoding network. When the original image and the three-dimensional object image are input into the encoding network for network coding, each sub-encoding network can output a spatial feature corresponding to a preset resolution. The encoding network for the original image and the three-dimensional object image can be the same or different encoding networks, but different encoding networks share network parameters. The structure of the encoding sub-network can be various, for example, it can be composed of a simple one-layer convolutional network, or it can be other encoding network structures. The preset resolution can be set according to the actual application, for example, the resolution can be from 4*4 to 512*512.

[0098] 105. The original spatial features, the three-dimensional spatial features and the preset basic style features are fused to obtain the target object image at the target resolution.

[0099] Among them, the target object is an object image in which the body posture of the original object is replaced with the body posture of the driving object. The so-called body posture can be understood as the posture information on the body of the object. For example, taking the object as a human body as an example, it can include the human body posture. Taking the object as a facial object, that is, a human face as an example, it can include facial posture, eyes and expressions, etc.

[0100] There are many ways to fuse the original spatial features, the three-dimensional spatial features, and the preset basic style features, which can be specifically as follows:

[0101] For example, based on the preset basic style features, a basic object image at an initial resolution can be generated, and the trained image generation model can be used to map the three-dimensional image features into latent features. Based on the latent features, the preset basic style features can be adjusted. According to the basic object image, the adjusted style features, original spatial features and three-dimensional spatial features can be fused to obtain the target object image at the target resolution.

[0102] The preset basic style features can be understood as the style features in the constant tensor (Const) pre-set during the image driving process. The so-called style features can be understood as the feature information used to generate an image of a specific style.

[0103] Among them, latent features can be understood as the intermediate features w that encode the three-dimensional image features. Different elements of the intermediate features w control different visual features, thereby reducing the correlation between features (decoupling and feature separation). The encoding process can be to extract the deep-level relationships hidden under the surface features of the three-dimensional image features, decouple these relationships, and thus obtain the latent features (latent code). There are many ways to use the trained image generation model to map the three-dimensional image features to latent features. For example, the mapping network of the trained image generation model can be used. Directly map 3D image features to latent features (w).

[0104] After mapping the latent features, the preset basic style features can be adjusted based on the latent features. The adjustment here can be understood as style modulation. There are many ways to adjust, for example, adjusting the size of the basic style features to obtain the initial style features, modulating the latent features to obtain the convolution weights corresponding to the initial style features, and adjusting the initial style features based on the convolution weights to obtain the adjusted style features.

[0105] The convolution weights can be understood as the weight information when convolving the initial style features. There are many ways to modulate the latent features. For example, the base convolution weights can be obtained and adjusted based on the latent features to obtain the convolution features corresponding to the initial style features. Adjusting the convolution weights based on latent features can be achieved primarily using the Mod and Demod modules in the decoding network of StyleGAN v2 (a style transfer model).

[0106] After the latent features are modulated, the initial style features can be adjusted based on the convolution weights obtained after the modulation process. There are many ways to adjust the style features. For example, in the style convolution network (StyleConv) of the trained image generation model, the target style convolution network corresponding to the resolution of the basic object image is selected, and the initial style features are adjusted based on the convolution weights to obtain the adjusted style features.

[0107] After adjusting the preset basic style features, the adjusted style features, original spatial features and three-dimensional spatial features can be fused according to the basic object image. There are many ways to fuse them. For example, based on the preset resolution, the target original spatial features are filtered out from the original spatial features, and the target three-dimensional spatial features are filtered out from the three-dimensional spatial features. The adjusted style features, the target original spatial features and the target three-dimensional spatial features are fused to obtain the fused style features at the current resolution. Based on the fused style features and the basic object image, the target object image at the target resolution is generated.

[0108] Among them, there are many ways to filter out the target original spatial features from the original spatial features based on the preset resolution, and filter out the target three-dimensional spatial features from the three-dimensional spatial features. For example, the original spatial features and the three-dimensional spatial features can be sorted separately based on the preset resolution, and based on the sorting information, the original spatial features with the smallest resolution can be filtered out from the original spatial features as the target original spatial features, and the original spatial features with the smallest resolution can be filtered out from the three-dimensional spatial features as the target original spatial features. After filtering out the target original spatial features and the target three-dimensional spatial features, the target original spatial features can be deleted from the original spatial features and deleted from the three-dimensional spatial features. In this way, the spatial features with the smallest resolution can be filtered out from the original spatial features and the three-dimensional spatial features each time, thereby obtaining the target original spatial features and the target three-dimensional spatial features.

[0109] After filtering out the target original spatial features and the target three-dimensional spatial features, the adjusted style features, the target original spatial features, and the target three-dimensional spatial features can be fused. There are many ways to fuse them. For example, the target original spatial features, the target three-dimensional spatial features, and the adjusted style features can be directly spliced ​​to obtain the fused style features at the current resolution. Specifically, it can be shown as formula (4):

[0110]

[0111] in, is the fused style feature, which can be the style feature corresponding to the next preset resolution of the basic style feature. As the basic style features, is the original spatial feature of the target at the preset resolution, is the target three-dimensional spatial feature at the preset resolution, Concat represents the concatenation or splicing of features, and StyleConv is the style convolutional network.

[0112] After obtaining the fused style features at the current resolution, the target object image at the target resolution can be generated based on the fused style features and the basic object image. There are many ways to generate the target object image at the target resolution. For example, the current object image can be generated based on the fused style features, and the current object image and the basic object image can be fused to obtain the fused object image at the current resolution. The fused style features are used as the preset basic style features, and the fused object image is used as the basic object image. The step of adjusting the preset basic style features based on the latent features is returned to execute until the current resolution is the target resolution, and the target object image is obtained.

[0113] In the process of generating the target object image at the target resolution, it can be found that the current object image at different resolutions is sequentially superimposed with the base object image. Moreover, during the superposition process, the resolution is successively increased, thereby outputting a high-definition target object image. Taking the target object image as an example, the target human image generated by this solution is more realistic (no uncanny valley effect) and more efficient than the digital human image driven by CG modeling.

[0114] Optionally, a basic optical flow field at the initial resolution can be generated based on the basic style features, thereby outputting a target optical flow field at the target resolution. The basic optical flow field can be understood as a visual field indicating the movement of key points in the object image at the initial resolution. There are many ways to output the target optical flow field. For example, a basic optical flow field at the initial style resolution can be generated based on the basic style features. Based on the basic optical flow field, the adjusted style features, original spatial features, and three-dimensional spatial features are fused to obtain the target optical flow field at the target resolution.

[0115] Among them, there are many ways to fuse the adjusted style features, original spatial features, and three-dimensional spatial features. For example, based on a preset resolution, the target original spatial features can be screened out from the original spatial features, and the target three-dimensional spatial features can be screened out from the three-dimensional spatial features. The adjusted style features and the target style features are fused to obtain the fused style features at the current resolution. Based on the fused style features and the basic optical flow field, the target optical flow field at the target resolution is generated.

[0116] Among them, there are many ways to generate a target optical flow field at the target resolution based on the fused style features and the basic optical flow field. For example, the current optical flow field can be generated based on the fused style features, and the current optical flow field and the basic optical flow field can be fused to obtain the fused optical flow field at the current resolution. The fused style features are used as the preset basic style features, and the fused optical flow field is used as the basic optical flow field. The step of adjusting the preset basic style features based on the latent features is returned to execute until the current resolution is the target resolution to obtain the target optical flow field.

[0117] Among them, the target object image and the target optical flow field can be generated simultaneously based on the preset basic style features, or the target object image or the target optical flow field can be generated separately based on the preset style features. Taking the simultaneous generation of the target object and the target optical flow field as an example, the preset basic style features, the basic object image and the basic optical flow field can be processed by the decoding network of the trained image generation model. Each decoding network can include a decoding sub-network corresponding to each preset resolution. The resolution can be increased from 4*4 to 512*512. The network structure of the decoding sub-network can be as follows: Figure 5As shown, the fused style features output by the previous decoding sub-network are received Fusion object image (I i ) and fused optical flow field (f i ), the latent feature w is modulated to obtain The corresponding convolution weight, and based on the convolution weight, Perform convolution processing to obtain the adjusted style features Based on the resolution corresponding to the decoding sub-network, the target original spatial features corresponding to the decoding sub-network are selected from the original spatial features. And filter out the target three-dimensional spatial features corresponding to the decoding sub-network in the three-dimensional spatial features Will and By connecting them in series, we can get the fused style features output by the decoding sub-network Then, based on Generate the current object image and the current optical flow field, and combine the current object image and the I output of the previous decoding sub-network i By fusing, we can get the fused object image (I i+1 ), the fused optical flow field (f i ) are fused to obtain the fused optical flow field (f i+1 ), and then output it to the decoding sub-network of the next layer until the decoding sub-network corresponding to the target resolution outputs the fused object image and the fused optical flow field. The fused object image at the target resolution can be used as the target object image, and the fused optical flow field at the target resolution can be used as the target optical flow field.

[0118] Optionally, the trained image generation model can be set according to actual applications. In addition, it should be noted that the trained image generation model can be pre-set by maintenance personnel or trained by the image processing device itself. That is, before the step of "using the trained image generation model to map the three-dimensional image features into latent features", the image processing method can further include:

[0119] Obtain video samples, and select any two frames of video frames from the video samples as original image samples and driving image samples respectively. Use the preset image generation model to drive the original image samples and the driving image samples to obtain the predicted object image. Based on the predicted object image and the driving image samples, converge the preset image generation model to obtain the trained image generation model. The specific method can be as follows:

[0120] (1) Obtain video samples and select any two video frames from the video samples as original image samples and driving image samples respectively.

[0121] There are many ways to obtain video samples, which can be as follows:

[0122] For example, a video sample uploaded by a terminal can be obtained, or at least one candidate video can be obtained from a video database or the Internet, and any one video can be screened out from the candidate videos as a video sample. Alternatively, at least one candidate video can be obtained from a video database or the Internet, and a video including an object can be screened out from the candidate videos to obtain a video sample.

[0123] After obtaining the video sample, any two frames of video frames can be selected from the video sample as the original image sample and the driving image sample respectively. There are many ways to select the original image sample and the driving image sample. For example, the video sample can be frame-processed to obtain a video frame set corresponding to the video sample, and any two frames of video frames can be selected from the video frame set, and the two frames of video frames can be used as the original image sample and the driving image sample respectively.

[0124] (2) Use the preset image generation model to perform object driving on the original image samples and the driving image samples to obtain the predicted object image.

[0125] For example, sample object features can be extracted from the original image samples, sample posture features can be extracted from the driving image samples, the sample object features and the sample posture features can be fused to obtain three-dimensional image sample features, and based on the three-dimensional image sample features, three-dimensional object image samples can be constructed. Spatial features at each preset resolution can be extracted from the original image samples and the three-dimensional object image samples to obtain the original spatial sample features of the original image samples and the three-dimensional spatial sample features of the three-dimensional object image samples. The original spatial sample features, the three-dimensional spatial sample features and the preset basic style features can be fused to obtain the predicted object image at the target resolution.

[0126] (3) Based on the predicted object image and the driving image samples, the preset image generation model is converged to obtain the trained image generation model.

[0127] For example, the predicted object image and the driving image samples are detected respectively, and based on the detection results, the adversarial loss information of the preset image generation model is determined; based on the predicted object image and the driving image samples, the reconstruction loss information of the preset image generation model is determined; the adversarial loss information and the reconstruction loss information are fused; and based on the fused loss information, the preset image generation model is converged to obtain the trained image generation model.

[0128] Among them, the adversarial loss information is the loss information of the adversarial network in judging the driving image sample and the predicted object image. There are many ways to detect the predicted object image and the driving image sample. For example, the discriminator D of the trained image generation model can be used to distinguish the driving image sample GT and the predicted object image respectively, and judge that the generated result is false and the real image is true, thereby obtaining the discrimination result and the detection probability of the discrimination result, and the discrimination result and the detection probability are used as the detection result.

[0129] After detecting the predicted object image and the driving image samples, the adversarial loss information of the preset image generation model can be determined based on the detection results. There are many ways to determine the adversarial loss information. For example, the detection probability of each discrimination result can be extracted from the detection results, and the adversarial parameters of the predicted object image and the driving image samples can be determined based on the detection probability. The adversarial parameters are then fused to obtain the adversarial loss information, which can be specifically shown in formula (5):

[0130]

[0131] Among them, L GAN To counter the loss information, D(GT) is the detection result of the driving image sample, D(G(input)) is the detection result of the predicted object image, GT is the driving image sample, input is the input original image sample and driving image sample, D is the discriminator, and G is the driving network (generating network).

[0132] Among them, the reconstruction loss information can be understood as the loss information between the reconstructed predicted object image and the driving image. There are many ways to determine the reconstruction loss information. For example, the image similarity of the predicted object image and the driving image sample can be calculated to obtain the perceptual loss information of the preset image generation model, and the predicted object image and the driving image can be compared to obtain the image loss information of the preset image generation model. The perceptual loss information and the image loss information can be fused to obtain the reconstruction loss information of the preset image generation model.

[0133] Among them, the perceptual loss information is used to indicate the image similarity between the predicted object image and the driving image sample. There are many ways to calculate the perceptual loss information. For example, LPIPS loss (a perceptual loss function) can be used to calculate the image similarity between the predicted object image and the driving image sample, and the image similarity can be used as the perceptual loss information.

[0134] The image loss information is used to indicate the feature distance between the image features of the predicted object image and the driving image sample. There are multiple ways to compare the predicted object image and the driving image sample. For example, the object image features can be extracted from the predicted object image, and the driving image features can be extracted from the driving image sample. The feature distance between the object image features and the driving image can be calculated to obtain the image loss information of the preset image generation model. There are multiple ways to calculate the feature distance. For example, L1 loss can be used to calculate the feature distance between the object image features and the driving image features to obtain the image loss information of the preset image generation model.

[0135] After calculating the perception loss information and the image loss information, the perception loss information and the image loss information can be fused to obtain the reconstruction loss information. There are many ways to fuse them. For example, the perception loss information and the image loss information can be directly added to obtain the reconstruction loss information. Specifically, it can be shown in formula (6):

[0136] L rec =|G(input)-GT|1+|LPIPS(G(input))-LPIPS(GT)|1 (6)

[0137] Among them, L rec To reconstruct the loss information, GT is the driving image sample, input is the input original image sample and driving image sample, G is the driving network (generating network), and |LPIPS(G(input))-LPIPS(GT)|1 is the perceptual loss information.

[0138] After the reconstruction loss information and the adversarial loss information are determined, the reconstruction loss information and the adversarial loss information can be fused. There are many ways to fuse them. For example, the reconstruction loss information and the adversarial loss information can be directly added together to obtain the fused loss information, which can be specifically shown in formula (7):

[0139] L=L GAN +L rec (7)

[0140] Among them, L is the loss information after fusion, L GAN To combat the loss of information, L rec To reconstruct the loss information.

[0141] Alternatively, it is also possible to obtain weighting parameters of the reconstruction loss information and the adversarial loss information, and based on the weighting parameters, weight the reconstruction loss information and the adversarial loss information respectively to obtain weighted reconstruction loss information and weighted adversarial loss information, and fuse the weighted reconstruction loss information and the weighted adversarial loss information to obtain fused loss information.

[0142] After obtaining the fused loss information, the preset image generation model can be converged based on the fused loss information to obtain the trained image generation model. There are many ways to converge the preset image generation model. For example, a gradient descent algorithm can be used to update the network parameters of the preset image generation model based on the fused loss information to converge the preset image generation model, thereby obtaining the trained image generation model. Alternatively, other convergence algorithms can be used to update the network parameters of the preset image generation model based on the fused loss information to obtain the trained image generation model.

[0143] Among them, when training the preset image generation model, the overall training process can be as follows Figure 6 As shown, two frames are randomly selected from the same video, one of which is used as the original image sample (I s ), one frame is used as a driving image sample (I d ), and this frame of driving image is also used as the target generation result (GroudTruth), thus obtaining training data. After obtaining the training data, the process of training the preset image generation model is generally divided into two parts, one is 3D reconstruction, and the other is to generate the predicted object image based on the generation network (driving network). In the 3D reconstruction part, ResNet50 is used as the 3D coefficient prediction network, and the ResNet50 network is used to obtain the original image sample (I s ) extracts identity, light, and texture information from the driving image sample (I d ) extracts the posture, and reconstructs the 3D image through the SMPL method based on the combination of these 3D coefficients. The texture and shadow of the 3D image come from the original image sample, and the posture and expression come from the driving image sample. In the generation network part, it is mainly based on Styleganv2 and includes the encoding network (Enc Block), the mapping network The encoding network extracts multi-layer spatial features from the 3D image and the driving image samples respectively. The output resolution is 4*4 at the lowest and 512*512 at the highest. The decoding module can match the output features one by one. The encoding module consists of a simple convolutional network. Mapping network The combination of 3D coefficients is mapped into latent features w in the latent space. The spatial features corresponding to each preset resolution obtained from the spatial encoding are decoded through the decoding network. During the decoding process, basic style features, basic object images, and basic optical flow fields are generated based on the constant tensor (const). The basic object images are then superimposed with the current object images, depending on the resolution from small to large, to obtain the predicted object image at the target resolution. The preset image generation model is converged based on the predicted object images and driving image samples to obtain a trained image generation model. When training the preset image generation model, the generative network (mapping network and decoding network) and discriminator in the preset image generation model can be pre-trained, while the encoding network needs to be trained from scratch. Therefore, during training, the learning rates of these three networks are different. The ratio of the learning rates can be set according to the actual application. For example, the learning rate ratio of the encoding network, generative network, and discriminator can be 100:10:1 or other ratios.

[0144] The trained image generation model can output the target object image and the target optical flow field simultaneously, or can output the target object image or the target optical flow field separately. Taking the trained image generation model outputting the target object image as an example, the process of outputting the target object image at the target resolution through the trained image generation model can be as follows: Figure 7 As described above, the body posture of the target image output by the trained image generation model is consistent with the driving image, and the texture is consistent with the original image. Figure 6 A comparison shows that the biggest difference between the usage process and the training process of the trained image generation model is that during usage, the input original image and driving image can be any object images, and the objects in the original image and the driving image can be the same object or different objects. During the training process, in order to improve the training accuracy of the model, different video frames in the same video are usually used as original image samples and driving image samples respectively.

[0145] From the above, it can be seen that after obtaining the original image of the original object and the driving image of the driving object, the embodiment of the present application extracts the object features of the original object in the original image and extracts the posture features of the driving object in the driving image. Then, the object features and the posture features are fused to obtain three-dimensional image features, and based on the three-dimensional image features, a three-dimensional object image is constructed. Then, spatial features at each preset resolution are extracted from the original image and the three-dimensional object image to obtain the original spatial features of the original image and the three-dimensional spatial features of the three-dimensional object image. Then, the original spatial features, the three-dimensional spatial features and the preset basic style features are fused to obtain the target object image at the target resolution. Since the scheme can construct a three-dimensional object image based on the three-dimensional object features extracted from the original image and the driving image, it can accurately capture the layout posture details of the driving object, and generate the target object image at the target resolution through the spatial features of the driving image and the three-dimensional reconstructed image at different resolutions, thereby ensuring the resolution of the generated target object image, thereby improving the accuracy of image processing.

[0146] The method described in the above embodiment will be further described in detail below with examples.

[0147] In this embodiment, the image processing device is specifically integrated into an electronic device, an electronic device server, and both the original object and the driven object are a human body.

[0148] like Figure 8 As shown, an image processing method, the specific process is as follows:

[0149] 201. The server obtains an original image of the original human body and a driven image of the driven human body.

[0150] For example, the server can directly obtain the original image of the original human body and the driving image of the driving human body uploaded by the terminal, or it can extract any two frames of video frames containing the human body from the video, use one of the frames as the original image, and the human body in the original image as the original human body, and use the remaining frame as the driving image, and the human body in the driving image as the driving human body, or it can also filter out human body images containing human bodies from the image data, and filter out any two of the human body images as the original image and the driving image, and so on.

[0151] 202. The server extracts the body features of the original body from the original image, and extracts the posture features of the driving body from the driving image.

[0152] For example, the server can use a trained ResNet50 network from the image generation model to extract 3D coefficients from both the original image and the driving image. The 3D coefficients extracted from the original image are used as the original image features, and the 3D coefficients extracted from the driving image are used as the driving image features. Parameters such as identity, lighting, and texture are extracted from the original image features to obtain the original human body features. Parameters such as posture are filtered from the driving image features to obtain the posture features of the driving human body.

[0153] 203. The server fuses the human body features and posture features to obtain three-dimensional image features.

[0154] For example, the server can directly splice the human body features and posture features to obtain three-dimensional image features, or it can also obtain weighted parameters of the human body features and posture features, and based on the weighted parameters, weight the human body features and posture features respectively to obtain weighted human body features and weighted posture features, and then fuse the weighted human body features and weighted posture features to obtain three-dimensional image features.

[0155] 204. The server constructs a three-dimensional human body image based on the three-dimensional image features.

[0156] For example, the server can extract the position information of at least one key point from the three-dimensional image features, convert the key point position information into geometric features, extract the texture information of the original human body from the three-dimensional image features, and extract texture features from the texture information. The specific conversion process can be shown in formula (2). Based on the geometric features and texture features, the three-dimensional model parameters of the three-dimensional human body are determined. Based on the three-dimensional model parameters, the SMPL reconstruction method is used to reconstruct the three-dimensional human body model. The three-dimensional human body model is projected onto a two-dimensional plane to obtain a three-dimensional human body image. The specific process can be shown in formula (3).

[0157] 205. The server extracts spatial features at each preset resolution from the original image and the three-dimensional human body image to obtain original spatial features of the original image and three-dimensional spatial features of the three-dimensional human body image.

[0158] For example, the server can use the encoding network (Enc Block) of the trained image generation model to spatially encode the original image and the three-dimensional human body image at each preset resolution, so as to obtain the original spatial features of the original image and the three-dimensional spatial features of the three-dimensional human body image at each resolution. The preset resolution can be 4*4 to 512*512.

[0159] 206. The server fuses the original spatial features, the three-dimensional spatial features, and the preset basic style features to obtain a target human body image at a target resolution.

[0160] For example, the server can generate a basic human image at an initial resolution based on the preset basic style features, and use the mapping network of the trained image generation model Directly map 3D image features to latent features (w).

[0161] The server resizes the basic style features to obtain initial style features. It then obtains basic convolution weights and adjusts them based on the latent features to obtain convolution features corresponding to the initial style features. The style convolution network (StyleConv) of the trained image generation model is then filtered to select a target style convolution network corresponding to the resolution of the basic human image. Based on the convolution weights, the initial style features are adjusted to obtain adjusted style features.

[0162] The server sorts the original spatial features and the three-dimensional spatial features based on a preset resolution. Based on the sorting information, the server selects the original spatial features with the lowest resolution from the original spatial features as the target original spatial features, and selects the original spatial features with the lowest resolution from the three-dimensional spatial features as the target original spatial features. After selecting the target original spatial features and the target three-dimensional spatial features, the target original spatial features can be deleted from the original spatial features and deleted from the three-dimensional spatial features. In this way, the spatial features with the lowest resolution can be selected from the original spatial features and the three-dimensional spatial features each time, thereby obtaining the target original spatial features and the target three-dimensional spatial features.

[0163] The server concatenates the target original spatial features, the target three-dimensional spatial features, and the adjusted style features to obtain the fused style features at the current resolution, which can be specifically shown in formula (4). Based on the fused style features, the server generates a current human body image, and fuses the current human body image with the base human body image to obtain a fused human body image at the current resolution. The fused style features are used as the preset base style features, and the fused human body image is used as the base human body image. The server then returns to the step of adjusting the preset base style features based on the latent features until the current resolution reaches the target resolution, thereby obtaining the target human body image.

[0164] Optionally, the server may also generate a basic optical flow field at the initial style resolution based on the basic style features. Based on the preset resolution, the target original spatial features are filtered out from the original spatial features, and the target three-dimensional spatial features are filtered out from the three-dimensional spatial features. The adjusted style features and the target style features are fused to obtain the fused style features at the current resolution. Based on the fused style features, the current optical flow field is generated, and the current optical flow field and the basic optical flow field are fused to obtain the fused optical flow field at the current resolution. The fused style features are used as the preset basic style features, and the fused optical flow field is used as the basic optical flow field. The server then returns to execute the step of adjusting the preset basic style features based on the latent features until the current resolution reaches the target resolution, thereby obtaining the target optical flow field.

[0165] The server may generate the target human image and the target optical flow field simultaneously based on the preset basic style features, or may generate the target human image or the target optical flow field separately based on the preset style features.

[0166] Optionally, before using the trained image generation model to map the 3D image features to latent features, the server may further train the preset image generation model to obtain the trained image generation model, which may be specifically as follows:

[0167] (1) The server obtains video samples and selects any two video frames from the video samples as the original image sample and the driving image sample respectively.

[0168] For example, the server can obtain a video sample uploaded by the terminal, or it can obtain at least one candidate video from a video database or the Internet, and filter out any video from the candidate videos as a video sample, or it can obtain at least one candidate video from a video database or the Internet, and filter out videos including human bodies from the candidate videos to obtain video samples.

[0169] (2) The server uses a preset image generation model to perform human body driving on the original image samples and the driving image samples to obtain a predicted human body image.

[0170] For example, the server can extract sample human features from the original image samples, extract sample posture features from the driving image samples, fuse the sample human features and the sample human features to obtain three-dimensional image sample features, and construct three-dimensional human image samples based on the three-dimensional image sample features, extract spatial features at each preset resolution from the original image samples and the three-dimensional human image samples, obtain the original spatial sample features of the original image samples and the three-dimensional spatial sample features of the three-dimensional human image samples, and fuse the original spatial sample features, the three-dimensional spatial sample features and the preset basic style features to obtain the predicted human body image at the target resolution.

[0171] (3) The server converges the preset image generation model based on the predicted human body image and driving image samples to obtain the trained image generation model.

[0172] For example, the server can use the discriminator D of the trained image generation model to discriminate the driving image sample GT and the predicted human body image respectively, and judge that the generated result is false and the real image is true, thereby obtaining the discrimination result and the detection probability of the discrimination result, and using the discrimination result and the detection probability as the detection result.

[0173] The server extracts the detection probability of each discrimination result from the detection results, and based on the detection probability, determines the adversarial parameters of the predicted human image and driving image samples, and fuses the adversarial parameters to obtain the adversarial loss information, which can be specifically shown in formula (5).

[0174] The server can use LPIPS loss to calculate the image similarity between the predicted human image and the driving image sample, and use the image similarity as the perceptual loss information. The human image features are extracted from the predicted human image, and the driving image features are extracted from the driving image sample. The feature distance between the human image features and the driving image is calculated, thereby obtaining the image loss information of the preset image generation model. There are many ways to calculate the feature distance. For example, L1 loss can be used to calculate the feature distance between the human image features and the driving image features, thereby obtaining the image loss information of the preset image generation model. By adding the perceptual loss information and the image loss information, the reconstruction loss information can be obtained, which can be specifically shown in formula (6). By adding the reconstruction loss information and the adversarial loss information, the fused loss information can be obtained, which can be specifically shown in formula (7).

[0175] The server uses a gradient descent algorithm to update the network parameters of the preset image generation model based on the fusion loss information to converge the preset image generation model, thereby obtaining a trained image generation model, or uses other convergence algorithms to update the network parameters of the preset image generation model based on the fusion loss information to obtain a trained image generation model.

[0176] Among them, the generation network (mapping network and decoding network) and discriminator in the preset image generation model can be pre-trained, while the encoding network needs to be trained from scratch. Therefore, during training, the learning rates of the three are different. The learning rate ratio of the encoding network, generation network and discriminator is 100:10:1.

[0177] From the above, it can be seen that after obtaining the original image of the original human body and the driving image of the driving human body, the server of the embodiment of the present application extracts the human body features of the original human body in the original image, and extracts the posture features of the driving human body in the driving image, and then fuses the human body features and posture features to obtain three-dimensional image features, and constructs a three-dimensional human body image based on the three-dimensional image features, and then extracts the spatial features at each preset resolution from the original image and the three-dimensional human body image to obtain the original spatial features of the original image and the three-dimensional spatial features of the three-dimensional human body image, and then fuses the original spatial features, the three-dimensional spatial features and the preset basic style features to obtain the target human body image at the target resolution; since the scheme can construct a three-dimensional human body image based on the three-dimensional human body features extracted from the original image and the driving image, it can accurately capture the layout posture details of the driving human body, and generate the target human body image at the target resolution through the spatial features of the driving image and the three-dimensional reconstructed image at different resolutions, thereby ensuring the resolution of the generated target human body image, and thus the accuracy of image processing can be improved.

[0178] The method described in the above embodiment will be further described in detail below with examples.

[0179] In this embodiment, description will be made by taking an example where the image processing device is specifically integrated into an electronic device and both the original object and the driven object are faces.

[0180] An image processing method, comprising:

[0181] An original facial image and a facial driving image are obtained, wherein the facial driving image is used as a template image for adjusting the facial posture in the original facial image, facial image features are extracted from the original facial image, and facial posture features are extracted from the facial driving image, the facial image features and the facial posture features are fused to obtain three-dimensional facial features, and a three-dimensional facial image is constructed based on the three-dimensional facial image, spatial features at each preset resolution are extracted from the original facial image and the facial driving image, original spatial features of the original facial image and three-dimensional spatial features of the three-dimensional facial image are obtained, the original spatial features, the three-dimensional spatial features and the preset basic style features are fused to obtain a target facial image at a target resolution, wherein the target facial image is an image in which the facial posture in the original facial image is replaced by the facial posture in the facial driving image.

[0182] like Figure 9 As shown, the specific process of the image processing method is as follows:

[0183] 301. Obtain an original facial image and a facial driving image.

[0184] The original facial image is the facial image whose facial posture needs to be adjusted, and the facial driving image is a template image used to adjust the facial posture in the original facial image. Adjusting the facial posture of the original facial image can be understood as replacing the facial expression, eye contact, and posture in the original facial image with the facial expression, eye contact, and posture of the facial driving image. Taking the human face as an example, the facial posture adjustment process can be as follows: Figure 10 shown.

[0185] There are many ways to obtain the original facial image and the facial drive image, which can be as follows:

[0186] For example, the original facial image and facial drive image uploaded by the terminal can be directly obtained, or any two video frames containing the face can be extracted from the video, one of which can be used as the original facial image and the remaining frame as the facial drive image, or facial images containing the face can be screened out from the image data, and any two of the facial images can be screened out as the original facial image and the facial drive image, etc.

[0187] 302. Extract facial image features from the original facial image, and extract facial posture features from the facial driving image.

[0188] Among them, facial image features are feature information used to characterize the identity, lighting, texture, etc. of the original facial image, and facial posture features are facial posture, gaze, and expression, etc. used to characterize the facial driving image.

[0189] For example, the image feature extraction network (ResNet50) of the trained image generation model can be used to extract image features from the original facial image and the facial driving image respectively to obtain original image features and driving image features. Feature information such as identity, light and shadow, and texture can be filtered out from the original image features as facial image features, and facial posture features can be filtered out from the driving image features.

[0190] The image feature extraction method can refer to the feature extraction method of the original image and the driving image, and will not be described in detail here.

[0191] 303. Fusing facial image features and facial posture features to obtain three-dimensional facial features, and constructing a three-dimensional facial image based on the three-dimensional facial image.

[0192] The three-dimensional facial features can be understood as image features for three-dimensional reconstruction of the face.

[0193] The three-dimensional facial image can be a facial image in which three-dimensional facial features are rendered into a three-dimensional facial model and projected onto a two-dimensional plane. It can also be understood as a facial 3D reconstruction. The reconstruction process can be as follows: Figure 11 shown.

[0194] There are many ways to fuse facial image features and facial posture features, which can be specifically as follows:

[0195] For example, facial image features and facial posture features can be directly spliced ​​to obtain three-dimensional facial features, or weighted parameters of facial image features and facial posture features can be obtained, and based on the weighted parameters, facial image features and facial posture features can be weighted respectively to obtain weighted facial image features and weighted facial posture features, and the weighted facial image features and weighted facial posture features can be fused to obtain three-dimensional facial features.

[0196] After fusing the facial image features and facial posture features, a three-dimensional facial image can be constructed based on the three-dimensional facial features obtained after the fusion. There are many ways to construct a three-dimensional facial image. For example, the three-dimensional facial features can be converted into geometric features and texture features of the three-dimensional face. Based on the geometric features and texture features, a three-dimensional facial model of the three-dimensional face can be constructed. The three-dimensional facial model can be projected onto a two-dimensional plane to obtain a three-dimensional facial image of the three-dimensional face.

[0197] The method of constructing a three-dimensional facial image can be referred to the process of constructing a three-dimensional object image, and will not be described in detail here.

[0198] 304. Extract spatial features at each preset resolution from the original facial image and the facial drive image to obtain original spatial features of the original facial image and three-dimensional spatial features of the three-dimensional facial image.

[0199] For example, the encoding network (Enc Block) of the trained image generation model can be used to spatially encode the original facial image and the three-dimensional facial image at each preset resolution, thereby obtaining the original spatial features of the original facial image and the three-dimensional spatial features of the three-dimensional facial image at each resolution.

[0200] The specific process of spatial encoding can be found in the method of spatial encoding the original image and the three-dimensional object image, and will not be described in detail here.

[0201] 305. Fusing the original spatial features, the three-dimensional spatial features, and the preset basic style features to obtain a target facial image at a target resolution.

[0202] The target facial image replaces the facial pose in the original facial image with the facial pose in the facial driving image. Facial pose can be understood as the state information of the facial region. For example, if the face is a human face, facial pose can be the face's eyes, expression, and facial posture.

[0203] There are many ways to fuse the original spatial features, the three-dimensional spatial features, and the preset basic style features, which can be specifically as follows:

[0204] For example, based on the preset basic style features, a basic object image at an initial resolution can be generated, and the trained image generation model can be used to map the three-dimensional image features into latent features. Based on the latent features, the preset basic style features can be adjusted, and the adjusted style features, original spatial features and three-dimensional spatial features can be fused according to the basic object image to obtain the target facial image at the target resolution.

[0205] Among them, the process of fusing the original spatial features, the three-dimensional spatial features and the preset basic style features to generate the target facial image of the target resolution can be referred to the process of generating the target object image at the target resolution, and will not be repeated here.

[0206] Optionally, a basic optical flow field at an initial resolution may be generated based on the basic style features, thereby outputting a target optical flow field at a target resolution. For details, please refer to the above description and will not be repeated here.

[0207] Optionally, the trained image generation model can be set according to actual applications. In addition, it should be noted that the trained image generation model can be pre-set by maintenance personnel or trained by the image processing device itself. The training process can be as follows:

[0208] Obtain video samples, and select any two frames of video frames from the video samples as original facial image samples and facial driven image samples respectively, use a preset image generation model to perform facial driving on the original facial image samples and facial driven image samples to obtain predicted facial images, and based on the predicted facial images and facial driven image samples, converge the preset image generation model to obtain a trained image generation model, which can be specifically as follows:

[0209] (1) Obtain a video sample and select any two frames from the video sample as the original facial image sample and the facial driven image sample respectively.

[0210] The method of obtaining the video sample can be referred to above and will not be described in detail here. After obtaining the video sample, the original facial image sample and the facial driven image sample can be screened out from the video sample. There are many ways to screen the original facial image sample and the facial driven image sample, which can be specifically as follows:

[0211] For example, the video sample is frame-processed to obtain a video frame set corresponding to the video sample, and any two video frames containing facial areas are screened out from the video frame set, and the two video frames are used as the original facial image sample and the facial driven image sample respectively.

[0212] (2) The preset image generation model is used to perform object driving on the original facial image samples and facial driving image samples to obtain the predicted object image.

[0213] For example, sample facial image features can be extracted from the original facial image samples, sample facial posture features can be extracted from the facial driving image samples, the sample facial image features and the sample facial posture features can be fused to obtain three-dimensional facial sample features, and based on the three-dimensional facial sample features, three-dimensional facial image samples can be constructed, and spatial features at each preset resolution can be extracted from the original image samples and the three-dimensional facial image samples to obtain the original spatial sample features of the original facial image samples and the three-dimensional spatial sample features of the three-dimensional facial image samples, and the original spatial sample features, the three-dimensional spatial sample features and the preset basic style features can be fused to obtain a predicted facial image at the target resolution.

[0214] (3) Based on the predicted facial image and facial driving image samples, the preset image generation model is converged to obtain the trained image generation model.

[0215] For example, the predicted facial image and facial driving image samples are detected respectively, and based on the detection results, the adversarial loss information of the preset image generation model is determined, based on the predicted facial image and facial driving image samples, the reconstruction loss information of the preset image generation model is determined, and based on the predicted facial image and facial driving image, the identity loss information of the preset image generation model is determined, the adversarial loss information, reconstruction loss information and identity loss information are fused, and the preset image generation model is converged based on the fused loss information to obtain the trained image generation model.

[0216] Among them, the method of determining the adversarial loss information and the reconstruction loss information based on the predicted facial image and the facial driving image samples can refer to the method of determining the adversarial loss information and the reconstruction loss information based on the predicted object image and the driving image samples. See above for details and will not be repeated here.

[0217] The identity loss information can be understood as the difference between the identity information of the reconstructed predicted facial image and the identity information of the facial driving image sample. There are many ways to determine the identity loss information of the preset image generation model based on the predicted facial image and the facial driving image sample. For example, facial identity features can be extracted from the predicted facial image sample, and facial driving identity features can be extracted from the facial driving image sample. The feature similarity between the facial identity features and the facial driving identity features is calculated, and based on the feature similarity, the identity loss information of the preset image generation model is determined. Specifically, it can be shown as formula (8):

[0218]

[0219] Among them, L ID For identity loss information, is the feature similarity, It is an identity (ID) feature extraction network. Its input is the original face image sample and the face driving image sample. GT is the face driving image sample and G is the driving network.

[0220] After the reconstruction loss information, the adversarial loss information, and the identity loss information are determined, the reconstruction loss information, the adversarial loss information, and the identity loss information can be fused. There are many ways to fuse them. For example, the reconstruction loss information, the adversarial loss information, and the identity loss information can be directly added together to obtain the fused loss information, which can be specifically shown in formula (9):

[0221] L=L GAN +L rec +L ID (9)

[0222] Among them, L is the loss information after fusion, L GAN To combat the loss of information, L rec To reconstruct the loss information, L ID Loss of identity information.

[0223] Alternatively, the weighting parameters of the adversarial loss information, reconstruction loss information and identity loss information can be obtained respectively, and based on the weighting parameters, the adversarial loss information, reconstruction loss information and identity loss information can be weighted respectively, and the weighted adversarial loss information, weighted reconstruction loss information and weighted identity loss information can be fused to obtain the fused loss information.

[0224] After obtaining the fused loss information, the preset image generation model can be converged based on the fused loss information. The convergence process can be seen in the above description and will not be described here in detail.

[0225] Taking the human face as an example, when training the preset image generation model, the training process can be as follows: Figure 12 As shown, two frames containing faces are randomly selected from the same video, and one frame is used as the original face image sample (I s ), one frame is used as a face driving image sample (I d ), and this frame of face driving image is also used as the target generation result (GroudTruth), thus obtaining training data. After obtaining the training data, the process of training the preset image generation model is generally divided into two parts: one is the 3D reconstruction of the face, and the other is to generate the predicted face image based on the generation network (driving network). In the 3D reconstruction of the face, ResNet50 is used as the 3D coefficient prediction network. The ResNet50 network is used to extract the face image sample (I s ) extracts identity, light, and texture information, and drives image samples from faces (I d ) extracts expressions, eyes, and posture, and reconstructs a 3D face image based on the combination of these 3D coefficients using the SMPL method. The identity, texture, and lighting of the 3D face image come from the original face image sample, while the posture, expression, and eyes come from the face driving image sample. In the generation network part, it is mainly based on Stylegan v2 and includes an encoding network (Enc Block), a mapping network The encoding network extracts multi-layer spatial features from 3D face images and face-driven image samples respectively. The lowest output resolution is 4*4 and the highest resolution is 512*512, which can match the output features of the decoding module one by one. The encoding module consists of a simple one-layer convolutional network. Mapping network The combination of 3D coefficients is mapped into latent features w in the latent space. The spatial features corresponding to each preset resolution generated by the spatial encoding are decoded through the decoding network. During the decoding process, basic style features, basic facial images, and basic optical flow fields are generated based on the constant tensor (const). The basic facial images are then superimposed with the current facial image, depending on the resolution from small to large, to obtain a predicted facial image at the target resolution. The preset image generation model is converged based on the predicted facial images and facial driving image samples to obtain a trained image generation model. When training the preset image generation model, the generative network (mapping network and decoding network) and discriminator in the preset image generation model can be pre-trained, while the encoding network needs to be trained from scratch. Therefore, during training, the learning rates of these three networks are different. The ratio of the learning rates can be set according to the actual application. For example, the learning rate ratio of the encoding network, generative network, and discriminator can be 100:10:1 or other ratios.

[0226] The trained image generation model can output the target face image and the target optical flow field simultaneously, or can output the target face image or the target optical flow field separately. Taking the trained image generation model outputting the target face image as an example, the process of outputting the target face image at the target resolution through the trained image generation model can be as follows: Figure 13 As mentioned above, the facial posture, expression and eyes of the target face image output by the trained image generation model are consistent with the face driving image, and the identity, light and shadow, texture and original face image are consistent. Figure 12 A comparison shows that the biggest difference between the usage process and the training process of the image generation model after training is that during usage, the input original face image and face-driven image can be face images of any object, and the objects in the original face image and the face-driven image can be the same object or different objects. During the training process, in order to improve the training accuracy of the model, different video frames in the same video are usually used as original face image samples and face-driven image samples respectively, and the objects in the original face image samples and the face-driven image samples are often the same object.

[0227] Optionally, in some embodiments, a current original image and a current driven image may be acquired, and region detection may be performed on the current original image and the current driven image. When it is detected that the current original image and the current driven image include a facial region and an object region, an image corresponding to the facial region may be segmented from the current original image to obtain an original facial image, and an image corresponding to the object region may be segmented from the original image to obtain an original image corresponding to the original object. An image corresponding to the facial region may be segmented from the current driven image to obtain a facial driven image, and an image corresponding to the object region may be segmented from the current driven image to obtain a driven image corresponding to the driven object. Then, an object image at a target resolution is generated based on the original image and the driven image, and a facial image at a target resolution is generated based on the original facial image and the facial driven image. The object image and the facial image are fused to obtain a target image at a target resolution, wherein the target image is an image in which the body posture and facial posture in the current original image are replaced with the body posture and facial posture in the driven image.

[0228] Among them, taking the facial area as the human face area and the torso area as the human body area as an example, the method of image driving according to the current original image and the current driving image can be regarded as combining human body driving and face driving, so as to generate a high-definition target image, which greatly increases the accuracy of image processing.

[0229] The process of generating an object image at a target resolution based on the original image and the driving image, and generating a facial image at a target resolution based on the original facial image and the facial driving image can be found above and will not be described in detail here.

[0230] From the above, it can be seen that after obtaining the original facial image and the facial driving image, the embodiment of the present application extracts facial image features from the original facial image and extracts facial posture features from the facial driving image. Then, the facial image features and the facial posture features are fused to obtain three-dimensional facial features, and a three-dimensional facial image is constructed based on the three-dimensional facial image. Then, spatial features at each preset resolution are extracted from the original facial image and the facial driving image to obtain the original spatial features of the original facial image and the three-dimensional spatial features of the three-dimensional facial image. The original spatial features, the three-dimensional spatial features and the preset basic style features are fused to obtain the target facial image at the target resolution. Since the scheme can construct a three-dimensional facial image based on the three-dimensional facial features extracted from the original facial image and the facial driving image, it can accurately capture details such as facial posture, expression and eye contact that drive the face, and generate a target facial image at the target resolution through the spatial features of the facial driving image and the three-dimensional reconstructed image at different resolutions, thereby ensuring the resolution of the generated target facial image, thereby improving the accuracy of image processing.

[0231] In order to better implement the above method, an embodiment of the present invention further provides an image processing device, which can be integrated into an electronic device, such as a server or a terminal. The terminal may include a tablet computer, a laptop computer and / or a personal computer.

[0232] For example, Figure 14 As shown, the image processing device may include an acquisition unit 401, a first extraction unit 402, a construction unit 403, a second extraction unit 404 and a fusion unit 405, as follows:

[0233] (1) Acquisition unit 401;

[0234] An acquisition unit 401 is configured to acquire an original image of an original object and a driving image of a driving object, wherein the driving image is used as a template image for adjusting the body posture of the original object;

[0235] For example, the acquisition unit 401 can be specifically used to obtain the original image of the original object and the driven image of the driven object uploaded by the terminal, or any two frames of video frames containing the object can be extracted from the video, one of the frames can be used as the original image, and the object in the original image can be used as the original object, and the remaining frame can be used as the driven image, and the object in the driven image can be used as the driven object, or the object image containing the object can be filtered out from the image data, and any two frames of the object image can be filtered out as the original image and the driven image, and so on.

[0236] (2) first extraction unit 402;

[0237] The first extraction unit 402 is configured to extract object features of the original object from the original image and extract posture features of the driven object from the driven image.

[0238] For example, the first extraction unit 402 can be specifically used to extract image features from the original image and the driving image respectively, to obtain the original image features of the original image and the driving image features of the driving image, to filter out the object features of the original object from the original image features, and to filter out the posture features of the driving object from the driving image features.

[0239] (3) Construction unit 403;

[0240] The construction unit 403 is configured to fuse the object features and the posture features to obtain three-dimensional image features, and construct a three-dimensional object image based on the three-dimensional image features.

[0241] For example, the construction unit 403 can be specifically used to fuse object features and posture features to obtain three-dimensional image features, convert the three-dimensional image features into geometric features and texture features of the three-dimensional object, construct a three-dimensional object model of the three-dimensional object based on the geometric features and texture features, and project the three-dimensional object model onto a two-dimensional plane to obtain a three-dimensional object image of the three-dimensional object.

[0242] (4) second extraction unit 404;

[0243] The second extraction unit 404 is configured to extract spatial features at each preset resolution from the original image and the 3D object image to obtain original spatial features of the original image and 3D spatial features of the 3D object image.

[0244] For example, the second extraction unit 404 can be specifically used to use the encoding network of the trained image generation model to spatially encode the original image and the three-dimensional object image at each preset resolution, thereby obtaining the original spatial features of the original image and the three-dimensional spatial features of the three-dimensional object image at each resolution.

[0245] (5) fusion unit 405;

[0246] The fusion unit 405 is used to fuse the original spatial features, the three-dimensional spatial features and the preset basic style features to obtain a target object image at a target resolution, wherein the target object image is an object image in which the body posture of the original object is replaced with the body posture of the driving object.

[0247] For example, the fusion unit 405 can be specifically used to generate a basic object image at an initial resolution based on preset basic style features, use the trained image generation model to map the three-dimensional image features into latent features, and adjust the preset basic style features based on the latent features, filter out the target original spatial features from the original spatial features based on the preset resolution, and filter out the target three-dimensional spatial features from the three-dimensional spatial features, fuse the adjusted style features, the target original spatial features and the target three-dimensional spatial features to obtain the fused style features at the current resolution, and generate the target object image at the target resolution based on the fused style features and the basic object image.

[0248] Optionally, in some embodiments, the image processing apparatus may further include a training unit 406, such as Figure 15 As shown, the specific details can be as follows:

[0249] The training unit 406 is used to train the preset image generation model to obtain a trained image generation model.

[0250] For example, the training unit 406 can be specifically used to obtain video samples, and select any two frames of video frames from the video samples as original image samples and driving image samples respectively, and use a preset image generation model to drive the original image samples and the driving image samples to obtain a predicted object image. Based on the predicted object image and the driving image samples, the preset image generation model is converged to obtain a trained image generation model.

[0251] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can be found in the previous method embodiments and will not be repeated here.

[0252] As can be seen from the above, in the embodiment of the present application, after the acquisition unit 401 acquires the original image of the original object and the driving image of the driving object, the first extraction unit 402 extracts the object features of the original object from the original image and extracts the posture features of the driving object from the driving image. Then, the construction unit 403 fuses the object features and the posture features to obtain three-dimensional image features, and constructs a three-dimensional object image based on the three-dimensional image features. Then, the second extraction unit 404 extracts spatial features at each preset resolution from the original image and the three-dimensional object image to obtain the original spatial features of the original image and the three-dimensional spatial features of the three-dimensional object image. Then, the fusion unit 405 fuses the original spatial features, the three-dimensional spatial features, and the preset basic style features to obtain a target object image at the target resolution. Since this solution can construct a three-dimensional object image based on the three-dimensional object features extracted from the original image and the driving image, it can accurately capture the layout posture details of the driving object, and generate a target object image at the target resolution through the spatial features of the driving image and the three-dimensional reconstructed image at different resolutions, thereby ensuring the resolution of the generated target object image, thereby improving the accuracy of image processing.

[0253] An embodiment of the present invention further provides an electronic device, such as Figure 16 , which shows a schematic structural diagram of an electronic device involved in an embodiment of the present invention, specifically:

[0254] The electronic device may include one or more processing core processors 501, one or more computer-readable storage media memories 502, a power supply 503, an input unit 504 and other components. Those skilled in the art will understand that Figure 16 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0255] The processor 501 is the control center of the electronic device. It connects all parts of the electronic device using various interfaces and circuits. It performs various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 502 and accessing data stored in the memory 502. Optionally, the processor 501 may include one or more processing cores. Preferably, the processor 501 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 501.

[0256] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0257] The electronic device also includes a power supply 503 for supplying power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 503 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0258] The electronic device may further include an input unit 504, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0259] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 501 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 502 according to the following instructions, and the processor 501 will run the application programs stored in the memory 502 to implement various functions as follows:

[0260] An original image of an original object and a driving image of a driving object are obtained, wherein the driving image is used as a template image for adjusting the body posture of the original object, object features of the original object are extracted from the original image, and posture features of the driving object are extracted from the driving image, the object features and the posture features are fused to obtain three-dimensional image features, and a three-dimensional object image is constructed based on the three-dimensional image features, spatial features at each preset resolution are extracted from the original image and the three-dimensional object image, original spatial features of the original image and three-dimensional spatial features of the three-dimensional object image are obtained, the original spatial features, the three-dimensional spatial features and the preset basic style features are fused to obtain a target object image at a target resolution, wherein the target object image is an object image in which the body posture of the original object is replaced with the body posture of the driving object.

[0261] For example, the electronic device obtains an original image of the original object and a driven image of the driven object uploaded by the terminal. Alternatively, it can extract any two video frames containing the object from the video, use one of the frames as the original image, and the object in the original image as the original object, and use the remaining frame as the driven image, and the object in the driven image as the driven object. Alternatively, it can filter out object images containing the object from the image data, and select any two of the object images as the original image and the driven image, respectively. Image feature extraction is performed on the original image and the driven image, respectively, to obtain original image features of the original image and driving image features of the driven image. Object features of the original object are filtered out from the original image features, and posture features of the driven object are filtered out from the driving image features. The object features and posture features are fused to obtain three-dimensional image features. The three-dimensional image features are converted into geometric features and texture features of the three-dimensional object. A three-dimensional object model of the three-dimensional object is constructed based on the geometric features and texture features. The three-dimensional object model is projected onto a two-dimensional plane to obtain a three-dimensional object image of the three-dimensional object. The encoding network of the trained image generation model is used to spatially encode the original image and the three-dimensional object image at each preset resolution, thereby obtaining the original spatial features of the original image and the three-dimensional spatial features of the three-dimensional object image at each resolution. Based on the preset basic style features, a basic object image at the initial resolution is generated. The trained image generation model is used to map the three-dimensional image features into latent features. Based on the latent features, the preset basic style features are adjusted. Based on the preset resolution, target original spatial features are selected from the original spatial features, and target three-dimensional spatial features are selected from the three-dimensional spatial features. The adjusted style features, target original spatial features, and target three-dimensional spatial features are fused to obtain fused style features at the current resolution. Based on the fused style features and the basic object image, a target object image at the target resolution is generated.

[0262] The specific implementation of the above operations can be found in the previous embodiments and will not be described in detail here.

[0263] From the above, it can be seen that after obtaining the original image of the original object and the driving image of the driving object, the embodiment of the present application extracts the object features of the original object in the original image and extracts the posture features of the driving object in the driving image. Then, the object features and the posture features are fused to obtain three-dimensional image features, and based on the three-dimensional image features, a three-dimensional object image is constructed. Then, spatial features at each preset resolution are extracted from the original image and the three-dimensional object image to obtain the original spatial features of the original image and the three-dimensional spatial features of the three-dimensional object image. Then, the original spatial features, the three-dimensional spatial features and the preset basic style features are fused to obtain the target object image at the target resolution. Since the scheme can construct a three-dimensional object image based on the three-dimensional object features extracted from the original image and the driving image, it can accurately capture the layout posture details of the driving object, and generate the target object image at the target resolution through the spatial features of the driving image and the three-dimensional reconstructed image at different resolutions, thereby ensuring the resolution of the generated target object image, thereby improving the accuracy of image processing.

[0264] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0265] To this end, an embodiment of the present invention provides a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps of any of the image processing methods provided in the embodiments of the present invention. For example, the instructions can execute the following steps:

[0266] An original image of an original object and a driving image of a driving object are obtained, wherein the driving image is used as a template image for adjusting the body posture of the original object, object features of the original object are extracted from the original image, and posture features of the driving object are extracted from the driving image, the object features and the posture features are fused to obtain three-dimensional image features, and a three-dimensional object image is constructed based on the three-dimensional image features, spatial features at each preset resolution are extracted from the original image and the three-dimensional object image, original spatial features of the original image and three-dimensional spatial features of the three-dimensional object image are obtained, the original spatial features, the three-dimensional spatial features and the preset basic style features are fused to obtain a target object image at a target resolution, wherein the target object image is an object image in which the body posture of the original object is replaced with the body posture of the driving object.

[0267] For example, an original image of an original object and a driven image of a driven object uploaded by a terminal are obtained. Alternatively, any two video frames containing an object can be extracted from a video, one of which is used as the original image, with the object in the original image as the original object, and the remaining frame as the driven image, with the object in the driven image as the driven object. Alternatively, object images containing the object can be screened from the image data, and any two of these object images can be screened as the original image and the driven image, respectively. Image feature extraction is performed on the original image and the driven image, respectively, to obtain original image features of the original image and driving image features of the driven image. Object features of the original object are screened from the original image features, and posture features of the driven object are screened from the driving image features. The object features and posture features are fused to obtain three-dimensional image features. The three-dimensional image features are converted into geometric features and texture features of the three-dimensional object. A three-dimensional object model of the three-dimensional object is constructed based on the geometric features and texture features. The three-dimensional object model is projected onto a two-dimensional plane to obtain a three-dimensional object image of the three-dimensional object. The encoding network of the trained image generation model is used to spatially encode the original image and the three-dimensional object image at each preset resolution, thereby obtaining the original spatial features of the original image and the three-dimensional spatial features of the three-dimensional object image at each resolution. Based on the preset basic style features, a basic object image at the initial resolution is generated. The trained image generation model is used to map the three-dimensional image features into latent features. Based on the latent features, the preset basic style features are adjusted. Based on the preset resolution, target original spatial features are selected from the original spatial features, and target three-dimensional spatial features are selected from the three-dimensional spatial features. The adjusted style features, target original spatial features, and target three-dimensional spatial features are fused to obtain fused style features at the current resolution. Based on the fused style features and the basic object image, a target object image at the target resolution is generated.

[0268] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0269] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0270] Since the instructions stored in the computer-readable storage medium can execute the steps in any image processing method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any image processing method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0271] According to one aspect of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the aforementioned image processing aspects or object-driven or face-driven aspects.

[0272] The above is a detailed introduction to an image processing method, device, electronic device and computer-readable storage medium provided by the embodiments of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. An image processing method, characterized in that: include: Acquire an original image of the original object and a driven image of the driven object, wherein the driven image is used as a template image for adjusting the body posture of the original object; extracting 3D coefficients from the original image and the driven image using the trained image generation model, and using the extracted 3D coefficients as object features of the original object and posture features of the driven object; fusing the object features and the posture features to obtain three-dimensional image features, constructing a three-dimensional object image based on the three-dimensional image features, and mapping the three-dimensional image features into latent features using the image generation model, wherein the latent features are intermediate features that encode the three-dimensional image features, and different elements of the intermediate features control different visual features; Extracting spatial features at each preset resolution from the original image and the three-dimensional object image to obtain original spatial features of the original image and three-dimensional spatial features of the three-dimensional object image; Based on preset basic style features, a basic object image is generated at an initial resolution, the three-dimensional image features are mapped into latent features using a trained image generation model, and the preset basic style features are adjusted based on the latent features; According to the basic object image, the adjusted style features, the original spatial features and the three-dimensional spatial features are fused to obtain a target object image at a target resolution, wherein the target object image is an object image in which the body posture of the original object is replaced with the body posture of the driving object.

2. The image processing method according to claim 1, wherein: The step of fusing the adjusted style features, the original spatial features, and the three-dimensional spatial features according to the basic object image to obtain a target object image at a target resolution includes: Based on the preset resolution, filtering out target original spatial features from the original spatial features, and filtering out target three-dimensional spatial features from the three-dimensional spatial features; fusing the adjusted style features, the target original spatial features, and the target three-dimensional spatial features to obtain a fused style feature at the current resolution; Based on the fused style features and the basic object image, a target object image at the target resolution is generated.

3. The image processing method according to claim 2, wherein: The step of generating the target object image at the target resolution based on the fused style features and the basic object image includes: generating a current object image based on the fused style features, and fusing the current object image with the base object image to obtain a fused object image at the current resolution; Using the fused style features as the preset basic style features, and using the fused object image as the basic object image; Return to the step of adjusting the preset basic style features based on the latent features until the current resolution reaches the target resolution, thereby obtaining the target object image.

4. The image processing method according to claim 1, wherein: The adjusting the preset basic style feature based on the latent feature includes: Adjusting the size of the basic style feature to obtain an initial style feature; Modulating the latent features to obtain convolution weights corresponding to the initial style features; Based on the convolution weight, the initial style feature is adjusted to obtain an adjusted style feature.

5. The image processing method according to claim 4, characterized in that The adjusting the initial style feature based on the convolution weight to obtain the adjusted style feature includes: Filtering a target style convolutional network corresponding to the resolution of the basic object image from the style convolutional network of the trained image generation model; Based on the convolution weights, the target style convolutional network is used to perform convolution processing on the initial style features to obtain adjusted style features.

6. The image processing method according to claim 1, wherein: Also includes: Based on the basic style features, generating a basic optical flow field at the initial resolution; According to the basic optical flow field, the adjusted style features, the original spatial features, and the three-dimensional spatial features are fused to obtain a target optical flow field at a target resolution.

7. The image processing method according to claim 1, wherein: Before mapping the three-dimensional image features into latent features using the trained image generation model, the method further includes: Obtaining a video sample, and selecting any two video frames from the video sample as an original image sample and a driving image sample respectively; Using a preset image generation model to perform object driving on the original image sample and the driving image sample to obtain a predicted object image; Based on the predicted object image and the driving image samples, the preset image generation model is converged to obtain a trained image generation model.

8. The image processing method according to claim 7, wherein: The step of converging the preset image generation model based on the predicted object image and the driving image sample to obtain a trained image generation model includes: Detecting the predicted object image and the driving image samples respectively, and determining adversarial loss information of the preset image generation model based on the detection results; Determining reconstruction loss information of the preset image generation model based on the predicted object image and the driving image sample; The adversarial loss information and the reconstruction loss information are fused, and the preset image generation model is converged based on the fused loss information to obtain a trained image generation model.

9. The image processing method according to claim 8, characterized in that: The determining, based on the predicted object image and the driving image sample, reconstruction loss information of the preset image generation model includes: Calculating the image similarity between the predicted object image and the driving image sample to obtain the perceptual loss information of the preset image generation model; Comparing the predicted object image with the driving image sample to obtain image loss information of the preset image generation model; The perceived loss information and the image loss information are fused to obtain reconstruction loss information of the preset image generation model.

10. The image processing method according to any one of claims 1 to 9, characterized in that: The constructing of a three-dimensional object image based on the three-dimensional image features includes: Converting the three-dimensional image features into geometric features and texture features of a three-dimensional object; constructing a three-dimensional object model of the three-dimensional object according to the geometric features and the texture features; The three-dimensional object model is projected onto a two-dimensional plane to obtain a three-dimensional object image of the three-dimensional object.

11. The image processing method according to any one of claims 1 to 9, characterized in that: The step of extracting the object features of the original object from the original image and extracting the posture features of the driven object from the driven image includes: performing image feature extraction on the original image and the driving image respectively to obtain original image features of the original image and driving image features of the driving image; Filtering out object features of the original object from the original image features; The posture features of the driving object are screened out from the driving image features.

12. An image processing method, characterized in that: include: Acquire an original facial image and a facial driving image, wherein the facial driving image is a template image for adjusting a facial posture in the original facial image; extracting 3D coefficients from the original facial image and the facial drive image using the trained image generation model, and using the extracted 3D coefficients as facial image features of the original facial image and facial posture features of the facial drive image; fusing the facial image features and facial posture features to obtain three-dimensional facial features, constructing a three-dimensional facial image based on the three-dimensional facial image, and mapping the three-dimensional facial features into latent features using the image generation model, wherein the latent features are intermediate features that encode the three-dimensional image features, and different elements of the intermediate features control different visual features; Extracting spatial features at each preset resolution from the original facial image and the facial drive image to obtain original spatial features of the original facial image and three-dimensional spatial features of the three-dimensional facial image; Based on preset basic style features, a basic object image is generated at an initial resolution, the three-dimensional image features are mapped into latent features using a trained image generation model, and the preset basic style features are adjusted based on the latent features; According to the basic object image, the adjusted style features, the original spatial features, and the three-dimensional spatial features are fused to obtain a target facial image at a target resolution, wherein the target facial image is an image in which a facial posture in the original facial image is replaced with a facial posture in the facial driving image.

13. An image processing device, characterized in that: include: an acquisition unit, configured to acquire an original image of the original object and a driving image of the driving object, wherein the driving image is used as a template image for adjusting the body posture of the original object; a first extraction unit, configured to extract 3D coefficients from the original image and the driven image using a trained image generation model, and use the extracted 3D coefficients as object features of the original object and posture features of the driven object; a construction unit, configured to fuse the object features and the posture features to obtain three-dimensional image features, construct a three-dimensional object image based on the three-dimensional image features, and map the three-dimensional image features into latent features using the image generation model, wherein the latent features are intermediate features that encode the three-dimensional image features, and different elements of the intermediate features control different visual features; a second extraction unit, configured to extract spatial features at each preset resolution from the original image and the three-dimensional object image, to obtain original spatial features of the original image and three-dimensional spatial features of the three-dimensional object image; A fusion unit is configured to generate a basic object image at an initial resolution based on preset basic style features, map the three-dimensional image features into latent features using a trained image generation model, adjust the preset basic style features based on the latent features, and fuse the adjusted style features, the original spatial features, and the three-dimensional spatial features according to the basic object image to obtain a target object image at a target resolution, wherein the target object image is an object image in which the body posture of the original object is replaced by the body posture of the driving object.

14. An electronic device, characterized in that: The system comprises a processor and a memory, wherein the memory stores an application program, and the processor is configured to run the application program in the memory to execute the steps of the image processing method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 12 are implemented.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the image processing method according to any one of claims 1 to 12.

Images