An image processing method, apparatus, device, medium, and program product

Abstract

This application discloses an image processing method, apparatus, device, medium, and program product. The method includes: acquiring an initial image containing a target object, and performing pose torsion processing on the target object in the initial image to obtain a pose torsion image; fusing the pose torsion image and the initial image to obtain a fused image; extracting a part image containing a specified portion of the target object from the fused image, and performing image super-resolution processing on the part image to obtain a super-resolution part image; and fusing the super-resolution part image and the fused image to obtain a target image. Using this application embodiment can improve the quality of the target object after pose transfer.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to the field of image processing, and more particularly to an image processing method, an image processing apparatus, an image processing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the rapid development of science and technology, human pose transfer technology has attracted widespread attention. Human pose transfer can be simply understood as the process of converting the pose of a target object contained in an image into the target's pose. Therefore, how to perform pose transfer on a target object to improve the quality of the transferred object has become a hot research topic. Summary of the Invention

[0003] This application provides an image processing method, apparatus, device, medium, and program product that can improve the quality of a target object after pose transfer.

[0004] On one hand, embodiments of this application provide an image processing method, the method comprising:

[0005] Obtain an initial image containing the target object, and perform pose torsion processing on the target object in the initial image to obtain a pose torsion image;

[0006] The pose torsion image and the initial image are fused to obtain a fused image; the pose of the target object in the fused image is the same as the pose of the target object in the pose torsion image.

[0007] Extract the part image containing the specified part of the target object from the fused image, and perform image super-resolution processing on the part image to obtain the super-resolution part image;

[0008] The target image is obtained by fusing the super-resolution region image and the fused image.

[0009] On the other hand, embodiments of this application provide an image processing apparatus, the apparatus comprising:

[0010] The acquisition unit is used to acquire an initial image containing the target object and perform pose torsion processing on the target object in the initial image to obtain a pose torsion image;

[0011] The processing unit is used to perform image fusion on the pose torsion image and the initial image to obtain a fused image; the pose of the target object in the fused image is the same as the pose of the target object in the pose torsion image.

[0012] The acquisition unit is used to extract a part image containing a specified part of the target object from the fused image, and to perform image super-resolution processing on the part image to obtain the super-resolution part image.

[0013] The processing unit is used to fuse the super-resolution part image and the fused image to obtain the target image.

[0014] In one implementation, the target object in the initial image presents an initial pose. The processing unit is used to perform pose torsion processing on the target object in the initial image to obtain a pose torsion image. Specifically, it is used for:

[0015] Acquire source images, which include source objects and the source objects are presented in the target pose;

[0016] The target object in the initial image is modeled to obtain the first model; the material object in the source image is modeled to obtain the second model; the model obtained by modeling any object includes: multiple meshes constituting the pose presented by any object, and each mesh is associated with the texture information of any object;

[0017] Based on the mesh correspondence between the first model and the second model, the texture information associated with each mesh in the first model is mapped to the corresponding mesh in the second model;

[0018] The mapped second model is used to generate an attitude torsion image.

[0019] In one implementation, when the processing unit acquires the source image, it specifically performs the following functions:

[0020] Obtain the source video, which consists of N image frames, where N is a positive integer.

[0021] Iterate through each image frame in the source video and determine the currently iterated image frame as the source image.

[0022] In one implementation, the processing unit is further used for:

[0023] After all the image frames in the source video have been traversed, N target images are obtained, and one target image corresponds to one image frame in the source video.

[0024] The N target images are arranged in order according to the playback order of the image frames corresponding to each target image in the source video to obtain the target video.

[0025] In one implementation, the processing unit is used to perform image fusion on the pose torsion image and the initial image to obtain the fused image, specifically by:

[0026] The target object in the initial image is subjected to feature extraction processing to obtain a first feature map; and the pose torsion image is subjected to feature extraction processing to obtain a second feature map; wherein each feature map includes multiple features;

[0027] Based on the feature correspondence between the first feature map and the second feature map, each feature in the first feature map is used to replace the corresponding feature in the second feature map;

[0028] Image reconstruction is performed based on the second feature map after feature replacement to obtain the fused image.

[0029] In one implementation, the processing unit is used to perform feature extraction processing on the target object in the initial image to obtain the first feature map, specifically for:

[0030] Perform foreground segmentation on the initial image to obtain a foreground image containing the target object;

[0031] Feature extraction is performed on the foreground image to obtain the first feature map.

[0032] In one implementation, the processing unit is used to perform image reconstruction based on the second feature map after feature replacement, and when obtaining the fused image, it is specifically used for:

[0033] Based on the second feature map after feature replacement, a color mode image containing the target object presenting the target pose and a mask image containing the target object presenting the target pose are reconstructed.

[0034] A fused image is generated based on the color mode image and the mask image.

[0035] In one implementation, when the processing unit generates the fused image based on the color mode image and the mask image, it specifically performs the following:

[0036] Obtain the target background image, which may include: the background image of the initial image, the background image of the source image, or a custom background image;

[0037] The color mode image and the mask image are fused to obtain an intermediate image;

[0038] The target background image is superimposed onto the intermediate image to obtain the fused image.

[0039] In one implementation, when the target background image is a custom background image, the processing unit is further configured to:

[0040] Display a data recommendation interface, which includes one or more multimedia data.

[0041] When any multimedia data in the data recommendation interface is selected, the target background image is segmented from the selected multimedia data.

[0042] The selected multimedia data is either an image or a video stream. When the selected multimedia data is a video stream, the target background image is segmented from any image frame in the selected multimedia data.

[0043] In one implementation, the processing unit performs image super-resolution processing on the part image to obtain the super-resolution part image, specifically by:

[0044] An image containing a specified part of the target object is extracted from the initial image and used as a reference image; wherein, the reference image has higher resolution than the image of the part.

[0045] Feature extraction is performed on the reference image to obtain reference image features, and feature extraction is performed on the part image to obtain part image features;

[0046] The features of the reference image and the feature image are fused, and the image is reconstructed based on the fused features to obtain the super-resolution feature image.

[0047] On the other hand, embodiments of this application provide an image processing apparatus, which includes:

[0048] A processor is used to load and execute computer programs;

[0049] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image processing method.

[0050] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the above-described image processing method.

[0051] On the other hand, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. The processor of an image processing device reads the computer instructions from the computer-readable storage medium, and when executed by the processor, the computer instructions implement the image processing method described above.

[0052] In this embodiment, the target object in the acquired initial image can be subjected to pose torsion processing, so that the distorted pose image contains the target object and the target object presents the target pose. Then, the initial image with richer details is used to fuse the pose torsion image with coarser details, resulting in a fused image with richer details. This achieves refinement of the target object presenting the target pose contained in the pose torsion image. Furthermore, since the target object contained in the pose torsion image already presents the target pose, the image fusion process can focus on enriching the detailed features of the pose torsion image without considering the pose torsion of the target object, reducing the difficulty of image fusion. In addition, it also supports image super-resolution processing on the part image containing a specified part of the target object extracted from the fused image, so that the specified part contained in the super-resolution part image has higher clarity, and image super-resolution processing of the part image containing the specified part can save image processing resources. Then, the super-resolution part image is fused with the fused image to obtain a target image with higher clarity. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1a This application illustrates a schematic diagram of the architecture of an image processing system provided in an exemplary embodiment.

[0055] Figure 1b This illustration shows a scene diagram of an image processing scenario provided by an exemplary embodiment of this application;

[0056] Figure 2 A schematic flowchart of an image processing method provided in an exemplary embodiment of this application is shown;

[0057] Figure 3 This illustration shows a schematic diagram of an initial image provided in an exemplary embodiment of this application;

[0058] Figure 4a This illustration shows a schematic diagram of an interface process for acquiring an initial image according to an exemplary embodiment of this application;

[0059] Figure 4b This illustration shows a schematic diagram of an interface flow for determining an initial image from a social conversation interface, provided by an exemplary embodiment of this application.

[0060] Figure 5This illustration shows a schematic diagram of an interface process for performing pose torsion processing on a target object in an initial image based on a source image, thereby obtaining a pose torsion image, according to an exemplary embodiment of this application.

[0061] Figure 6a A schematic diagram of a material selection interface provided in an exemplary embodiment of this application is shown;

[0062] Figure 6b A schematic diagram of a material acquisition interface provided in an exemplary embodiment of this application is shown;

[0063] Figure 6c This illustration shows a schematic diagram of obtaining an initial image and source images from the same interface, according to an exemplary embodiment of this application.

[0064] Figure 7 This illustration shows a schematic diagram of a fused image provided in an exemplary embodiment of this application;

[0065] Figure 8 This illustration shows a schematic diagram of a region image obtained after performing image super-resolution processing on a region image according to an exemplary embodiment of this application;

[0066] Figure 9 This illustration shows a schematic diagram of a target image provided in an exemplary embodiment of this application;

[0067] Figure 10 A schematic flowchart of an image processing method provided in an exemplary embodiment of this application is shown;

[0068] Figure 11 This illustration shows a schematic diagram of modeling a target object and a material object according to an exemplary embodiment of this application;

[0069] Figure 12 A schematic diagram of a mesh mapping provided in an exemplary embodiment of this application is shown;

[0070] Figure 13a This illustration shows a schematic diagram of the structure of a GAN generative network provided in an exemplary embodiment of this application;

[0071] Figure 13b This illustration shows a schematic diagram of generating a fused image based on a color mode image and a mask image, according to an exemplary embodiment of this application.

[0072] Figure 14 This illustration shows a schematic diagram of a foreground segmentation process for an image, which yields a background image and a foreground image, according to an exemplary embodiment of this application.

[0073] Figure 15This illustration shows a schematic diagram of an exemplary embodiment of the present application, which uses a background image of a source image as a target background image and overlays it with an intermediate image to obtain a fused image;

[0074] Figure 16 This illustration shows a schematic diagram of a target background image that is a user-defined background image, as provided in an exemplary embodiment of this application.

[0075] Figure 17 This illustration shows a structural diagram of a face enhancement model provided in an exemplary embodiment of this application;

[0076] Figure 18 This invention provides a schematic diagram of the structure of an image processing apparatus according to an exemplary embodiment of the present application.

[0077] Figure 19 A schematic diagram of the structure of an image processing device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0078] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0079] This application embodiment relates to an image processing system, which can, as follows: Figure 1aAs shown, the image processing system may include a terminal and a server, and this application does not limit the number of terminals and servers. The terminal may include, but is not limited to, smartphones (such as Android phones, iOS phones, etc.), tablet computers, portable personal computers, mobile internet devices (MIDs), smart TVs, in-vehicle devices, head-mounted devices, and other touch-screen smart devices. The terminal may run applications with image processing capabilities (hereinafter referred to as applications); an application can refer to a computer program that performs one or more specific functions; according to the way applications run, applications may include, but are not limited to, clients installed on the terminal, small programs that can be used without downloading and installation, web applications opened through a browser, etc. According to the functional type of the application, applications may include, but are not limited to, IM (Instant Messaging) applications, content interaction applications, etc.; among them, instant messaging applications refer to applications based on the Internet for instant communication and social interaction, and instant messaging applications may include, but are not limited to, social applications with communication functions, map applications with social interaction functions, game applications, etc. Content interaction applications refer to applications that can realize content interaction, such as online banking, sharing platforms, personal spaces, news applications, etc. This application does not limit the specific type of application described above as the target application. The server can include, but is not limited to, devices with complex computing capabilities such as data processing servers, web servers, and application servers. The server can be a backend server for any application, used to interact with the terminal running that application to provide computing and application service support. The server can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers. The terminal and server can communicate directly or indirectly via wired or wireless means; this application does not limit the connection method between the terminal and the server.

[0080] Based on the aforementioned image processing system, this application proposes an image processing scheme that can achieve human pose transfer. Human pose transfer, also known as pose transfer technology or human pose transfer technique, is an image generation technology that converts the pose of a target object (such as a person or animal) contained in an image into a target pose (such as any specified pose). Specifically, given an initial image and a target pose, assuming the target human body (such as any human body) in the initial image is in the initial pose, pose transfer processing on the target human body can generate a target image. This target image contains the target human body from the initial image, and the target human body is in the target pose. The above-described human pose transfer conversion process can be found in [reference needed]. Figure 1a ,like Figure 1a As shown, assuming the image to be pose-transformed is the initial image 101, which contains the target human body 1011, and the target pose is the pose presented by the material human body 1021 in the source image 102; then, after performing human pose transfer on the target human body 1011 in the initial image 101, a target image 103 can be generated, which contains the target human body 1011, and the pose presented by the target human body 1011 matches (or is the same as) the target pose presented by the material human body 1021 in the source image 102. Human pose transfer technology can realize the pose transformation of the target object in the initial image. When multiple target poses exist, multiple frames of target images containing the same human body but presenting different target poses can be generated based on the initial image. This allows human pose transfer technology to be widely applied in various scenarios, such as scenarios where video streams are generated based on initial images, enriching image generation gameplay to a certain extent.

[0081] In a specific implementation, the image processing scheme can be executed by the target terminal (i.e., any terminal) in the aforementioned image processing system, or by a target application (such as any application) running on the target terminal that has image processing capabilities, or by the target terminal (or the target application running on the target terminal that has image processing capabilities) and the server jointly, or by the server itself. Specifically, the image processing scheme may include: acquiring an initial image containing a target object presenting an initial pose (such as...). Figure 1a The initial image 101 shown is used, along with the target pose (or a source image containing a source object presenting the target pose). The target object in the initial image undergoes pose torsion processing, resulting in a pose-torsion image containing the target object in the target pose. Then, relying on the rich details of the target object in the initial image, the pose-torsion image and the initial image are fused to obtain a more detailed fused image. Finally, details of specific parts of the target object in the fused image (such as the face) can be enhanced to obtain a more recognizable target image (such as...). Figure 1a The target image 102 is shown. In the above scheme, the pose of the target object is first twisted, which allows the subsequent image fusion process to focus on improving the resolution of the target object. Furthermore, using an initial image with richer details to refine the pose-twisted image with coarser details can enrich the details of the fused image and improve the clarity of the target object's texture in the fused image. In addition, it also supports image super-resolution processing of part images containing specified parts of the target object, resulting in higher clarity of the super-resolution part images. Moreover, performing image super-resolution processing on part images containing specified parts can save image processing resources. Then, the super-resolution part images are fused with the fused image to obtain a target image with higher clarity.

[0082] It is worth noting that when multiple frames of source images exist, the initial pose of the target object in a single initial frame can be processed according to the target pose presented by the source object in the multiple frames of source images to obtain the target image corresponding to each frame of source images. The target image corresponding to any source image contains the target object, and the pose presented by the target object is consistent with the target pose presented by the source object in any source image. Furthermore, if the multiple frames of source images are multiple image frames contained in the same source video, the multiple target images can be sorted according to the playback order of the multiple image frames in the source video to generate a target video. When the target video is played, each video frame contains the target object, and the target object in each video frame presents a different target pose.

[0083] To better understand the image processing scenarios mentioned above, the following will combine... Figure 1b This image processing scenario will be described in detail; such as Figure 1b As shown, firstly, an initial image 101 to be processed is acquired, which contains a target human body 1011 and the target human body 1011 is initially in a standing posture; and secondly, a source video 104 is acquired, which includes N image frames, where N is a positive integer, such as... Figure 1b The shown video source contains image frames 1041, 1042, ..., etc. Next, each image frame in the video source 104 is traversed, and the currently traversed image frame is determined as the source image. Thus, after all image frames in the video source 104 have been traversed, N target images are obtained, with each target image corresponding to one image frame in the video source 104. For example... Figure 1bThe shown material video 104 contains a material human body 1021. In image frame 1041 of material video 104, material human body 1021 is in a standing posture with arms outstretched, and in image frame 1041, material human body 1021 is in a heart-shaped gesture. Then, by performing the above image processing scheme on the initial image 101 and image frame 1041, a target image 1051 is obtained. The target image 1051 contains the target human body 1011 contained in the initial image 101, and the target human body 1011 presents the target posture (i.e., standing posture with arms outstretched) presented by material human body 1021 in image frame 1041. Furthermore, by performing the above image processing scheme on the initial image 101 and image frame 1042, a target image 1052 is obtained. The target image 1052 contains the target human body 1011 contained in the initial image 101, and the target human body 1011 presents the target posture (i.e., heart-shaped gesture) presented by material human body 1021 in image frame 1042. Finally, according to the playback order of the image frames corresponding to each target image in the source video 104, the N target images are arranged in sequence to obtain the target video 105. For example, according to the playback order of image frames 1041 and 1042 in the source video 104, the target images 1051 and 1052 are sorted to obtain the target video 105. When playing the target video 105, the playback order of target images 1051 and 1052 is: target image 1051 → target image 1052. Based on this, by adopting the image processing scheme mentioned in the embodiments of this application, it is possible to generate multiple target images containing the target human body and presenting the target human body in different poses based on a single frame of initial image, thereby improving the generation efficiency of target images.

[0084] It should be noted that since the image processing flow between each frame in the source video and the initial image is consistent, the following descriptions will use the example of performing a pose reversal on the initial image based on each frame in the source video to illustrate the image processing scheme. Furthermore, the above explanation uses a human body as the object to be reversed; in practice, the object could also be an animal (such as a cat or dog). For ease of explanation, the target object will be used to replace the target human body in the initial image described above, and the source object will be used to replace the source human body in the source images described above.

[0085] Based on the image processing scheme described above, this application proposes a more detailed image processing method. The image processing method proposed in this application will be described in detail below with reference to the accompanying drawings.

[0086] Figure 2This illustration shows a flowchart of an image processing method provided by an exemplary embodiment of this application. The embodiment is described using the example of the image processing method being executed by the aforementioned target terminal. This image processing method may include, but is not limited to, steps S201-S205:

[0087] S201: Obtain the initial image containing the target object.

[0088] The target object in the initial image is the object to be posed and twisted. This object can be a human or an animal (such as a puppy or kitten); the following descriptions will use a human as the target object. Specifically, the initial image may contain all or part of the target object's torso; for example... Figure 3 As shown in the first image, the initial image contains the entire torso of the target object 301 (from head to toe); and so on. Figure 3 As shown in the second figure, the initial image contains part of the torso of the target object 301, such as only including the torso from head to waist; this application embodiment does not limit whether the initial image specifically includes all or part of the torso of the target object, which is stated here.

[0089] In practical implementation, when a target user (such as a user of a target terminal) needs to perform human pose transfer on a target object in an initial image, they can open and use the target application to obtain the initial image, and then perform human pose transfer processing on the obtained initial image to obtain the pose-transferred target object. The target application can be a social application with social functions. This way, if there is a need for image processing of the initial image during social interaction within the social application, the social application can be used directly for image processing. Compared to jumping from a social application to other applications with image processing capabilities, this enriches the functionality of the social application, shortens the image processing path, and improves image processing efficiency. Alternatively, the target application can be a small program running within any application. By running a target application with image processing capabilities within other applications in this lightweight way, it is possible to quickly launch the target application within other applications to perform pose transfer processing on the initial image, improving image processing speed and efficiency.

[0090] The following is an exemplary implementation of obtaining an initial image through a target application. Specifically, an image acquisition interface is displayed, which includes an image acquisition entry point. When the image acquisition entry point is triggered, an image display window is displayed, containing one or more candidate images. When any candidate image is selected from the image display window according to an image selection operation, the selected candidate image is determined as the initial image. If there is a return operation to the image acquisition interface, the user returns from the image display window to the image acquisition interface, and the initial image is displayed in the image acquisition interface. An exemplary interface flow of the above implementation process can be found in [reference needed]. Figure 4a ,like Figure 4a As shown, the image acquisition interface 401 includes an image acquisition entry 4011. When the image acquisition entry 4011 is triggered (such as a single click, double click, long press, etc.), an image display window 402 can be displayed. The image display window 402 includes at least one candidate image, such as candidate image 4021, candidate image 4022, ... When any candidate image, such as candidate image 4021, is selected, the selected candidate image 4021 is determined as the initial image, and the image acquisition interface 401 is returned. At this time, the selected candidate image 4021 is displayed in the image acquisition interface 401, that is, the initial image is displayed.

[0091] It should be noted that the above description only provides an exemplary interface flow for obtaining the initial image; the method of obtaining the initial image may vary in actual application scenarios. For example: ① Divide the image acquisition entry point by... Figure 4a In addition to displaying the style shown, it can also be displayed as a button (or control, component). ② Figure 4a In addition to the image acquisition entry 401, the image acquisition interface 401 shown may also include an image recommendation area 4012. This image recommendation area 4012 is used to display at least one recommended image, allowing the target user to directly select a recommended image as the initial image from the image acquisition interface 401. Specific selection methods may include, but are not limited to: dragging any recommended image (such as recommended image 40121) from the image recommendation area 4012 to the display area occupied by the image acquisition entry 4011; or triggering any recommended image in the image recommendation area 4012, such as by clicking, long-pressing, or double-clicking any recommended image; and so on. The recommended images in the image recommendation area 4012 may be images stored in the target terminal's local memory (such as images in the target terminal's photo album), images pre-set by the target application, or images obtained from the Internet, etc.

[0092] ③ Divide the image display window 402 by Figure 4aIn addition to being displayed on top of the image acquisition interface 401, the image display window 402 can also be displayed as a separate interface or as a window within the image acquisition interface. This embodiment does not limit the specific display format of the image display window 402. ④ The image display window 402 contains multiple type options, each associated with at least one candidate image. For example, if the type option is a local image, it means that the candidate image associated with that type option is stored in the local space of the target terminal. Similarly, if the type option is an internet image, it means that the candidate image associated with that type option is obtained from the internet. Candidate images can be searched from the internet according to image acquisition rules, which may include, but are not limited to: acquiring internet images and selecting multiple frames from the top of the order based on image popularity (number of clicks, coins, favorites, etc.); or acquiring images from the internet based on the target user's historical behavior (such as likes, favorites, etc.); and so on.

[0093] In other implementations, if the target application is one of the aforementioned social applications, this embodiment also supports determining an image during a social conversation as the initial image. Specifically, a social message stream is displayed in the social conversation interface of the social application. This social message stream consists of one or more social conversation messages, each sent by a user participating in the social conversation. Assuming the social message stream includes a social message containing an image, when a triggering event for that social message is detected, an option bar is displayed. This option bar includes an image processing option. When the image processing option is selected, it indicates that the target user wants to perform human pose transfer on the image, and the image is then determined as the initial image. The triggering event for the social message may include, but is not limited to: events generated when a triggering operation (such as clicking, double-clicking, right-clicking, etc.) is performed on the social message in the social conversation interface; events generated when the existence of the social message is detected in the social conversation interface; events generated when a cursor (such as the cursor mapped on the screen by a device like an electronic pen, remote control, or mouse) is detected in the social conversation interface within the display area of ​​the social message. This embodiment does not limit which one or more triggering events are specifically selected.

[0094] Combination Figure 4b The specific implementation process of determining the image in the social conversation interface as the initial image, as described above, will be introduced, such as... Figure 4bAs shown, a social message 4031 containing an image is displayed in the social conversation interface 403. When a trigger event occurs for the social message 4031 (such as any of the events mentioned above), an option bar 4032 is displayed, containing at least one option, including a pose transformation option 4033. When the pose transformation option 4033 is selected, it indicates a desire to perform human pose transformation on the image contained in the social message 4031, and that image is then selected as the initial image. Optionally, an image acquisition entry can be included in the shortcut area 4034 of the social conversation interface, allowing the target user to trigger the operation of selecting the initial image through the shortcut area 4034.

[0095] It should be noted that the above description of obtaining the initial image from a social conversation interface is based on an example of a social conversation interface. This application embodiment also supports obtaining the initial image from social public service interfaces (such as Moments, Dynamics, etc.). For example, if an image is displayed in Moments, that image can be selected as the initial image. This method of directly obtaining the initial image from a social interface (such as a social conversation interface or a social public service interface) combines social functions and image processing functions, allowing the target user to perform image processing during social interaction. Compared to copying the image to other image processing applications, this improves image processing efficiency and speed. Furthermore, when the target application is a social application, i.e., in a social conversation scenario where the initial image is processed by human pose transfer to obtain the target image, this application embodiment also supports directly sharing the pose-reversed target image to a social interface, such as sharing the target image to the social conversation interface containing the initial image, or sharing the target image to Moments, etc. This improves the sharing efficiency of the target image and shortens the sharing path. In addition, the above implementation methods are all introduced using the selection of one initial image as an example. However, in practice, multiple initial images can be selected at once. This allows for subsequent human pose transfer processing on each image, rather than selecting one initial image for processing and then selecting another initial image for processing, which can improve image processing efficiency.

[0096] S202: Perform pose torsion processing on the target object in the initial image to obtain a pose torsion image.

[0097] Pose torsion processing of a target object can refer to twisting the target object's pose from its initial pose as presented in the initial image to the target pose, thus generating a pose torsion image in which the target object presents the target pose. Specifically, pose torsion processing can be performed on a source image to obtain a pose torsion image; wherein the source image contains a source object and the source object presenting the target pose; a description of the source object can be found in the aforementioned description of the target object, and will not be repeated here. An example image of a pose torsion processing of the initial pose of a target object in an initial image, based on the target pose presented by the source object in the source image, to obtain a pose torsion image can be found [link to example image]. Figure 5 ;like Figure 5 As shown, the source image 102 contains source object 501, and the target posture of source object 501 is a heart-shaped gesture. The initial image 101 contains target object 301, and the initial posture of target object 301 is a standing posture. Then, after performing posture twisting processing on the initial posture of target object 301 based on source object 501, posture twisting image 502 of target object 301 showing a heart-shaped gesture can be obtained.

[0098] In this implementation, the source images can be pre-defined by business personnel based on business needs. After obtaining an initial object containing the target object, the pre-defined source images can be used to directly perform pose torsion processing on the target object in the initial image to obtain a pose torsion image. Alternatively, the source images can be determined by the target user. This method, where the target user determines the source images, can include: displaying a source image selection interface containing one or more candidate source images; when any candidate source image is selected, the selected candidate source image is determined as the source image containing the target pose. A schematic diagram of an exemplary source image selection interface can be found [link to schematic diagram]. Figure 6a ,like Figure 6a As shown, in the material selection interface 601, a candidate material image can be selected as the material image containing the target pose; or, multiple candidate material images can be selected as the material images containing the target pose. When multiple candidate material images are selected as material images, the target object in the initial image can be subjected to pose torsion processing according to the target pose of the material object contained in each of the multiple material images, to obtain multi-frame pose torsion images of the target object presenting different target poses. The candidate material images included in the material selection interface 601 can be one or more from the local space of the target terminal, the Internet, or settings of business personnel; this embodiment of the application does not limit this.

[0099] This application embodiment supports triggering the display of the material selection interface from the material acquisition interface, and enables the selection of material images from the material selection interface; such as Figure 6bAs shown, the material acquisition interface 602 includes a material acquisition entry point 6021. When the material acquisition entry point 6021 is selected, the material selection interface 601 is displayed, which contains one or more candidate material images. The target user can select a material image from the material selection interface 601. If a return operation from the material selection interface 601 to the material acquisition interface 602 is detected, the user returns to the material acquisition interface 602, and the material image is displayed in the material acquisition interface 602. The style changes of each interface in the above process, as well as the display method of the candidate material images, can be referred to the aforementioned description of the specific implementation method for selecting the initial image, and will not be repeated here.

[0100] It is worth noting that the image acquisition entry point used to obtain the initial image and the material acquisition entry point used to obtain the source images can be set in different interfaces, such as setting the image acquisition entry point in... Figure 4a The image acquisition interface shown is located in [location]. Figure 6b The provided image acquisition interface is shown. However, this embodiment also supports setting the image acquisition entry and the material acquisition entry in the same interface, that is, acquiring the initial image and the material image on the same interface. In this implementation, when the target user needs to perform human pose transfer processing on the target object in the initial image, the selection of the initial image and the material image can be realized on the same interface. Compared with acquiring the initial image and the material image from different interfaces, it can avoid cumbersome interface jumps, shorten the image acquisition path, and improve image acquisition efficiency.

[0101] An exemplary interface diagram illustrating the acquisition of initial and source images from the same interface can be found here. Figure 6c ,like Figure 6c As shown, the image interface 603 includes an image acquisition entry 4011 for acquiring an initial image and a material acquisition entry 6021 for acquiring source images. When any acquisition entry is selected, the interface associated with that acquisition entry can be triggered to display. For example, when the image acquisition entry 4011 is selected, the interface associated with that acquisition entry can be triggered to display... Figure 4a The image display window shown allows the target user to select an initial image. After the target user selects the initial image, they can directly return to the image interface 603, which contains the selected initial image. The target user can then trigger the material acquisition entry 6021 in the image interface 603 to select a material image. This embodiment does not limit the execution order of acquiring the initial image and acquiring the material image; that is, the target user can choose which image acquisition step to perform first according to their preference.

[0102] It is understandable that the source image can be any frame from the multiple frames contained in the source video; therefore... Figure 6a , Figure 6b as well as Figure 6c The icons for all or part of the candidate source images can be replaced with icons for candidate videos. When any candidate video is selected, it indicates that the target user wants to twist the initial pose of the target object in the initial image to the target pose presented by the source object in the candidate video. Since the candidate video contains at least two frames, twisting the initial image based on the candidate video can yield at least two target images. These at least two target images are sorted according to the playback order of the frames in the candidate video to obtain the target video. When the target video is played, the target object performs the actions performed by the source object in the source video. Optionally, this embodiment also supports the target user selecting a frame from the candidate video as a source image. For example, when any candidate video is selected, the playback interface of the candidate video can be displayed. The playback interface includes a progress bar and a pointer indicated by the progress bar (e.g., the pointer is on the progress bar). The target user can select a source image by dragging the pointer; the image pointed to by the pointer is then determined as the source image. Based on this, this embodiment enriches the target user's choice of source images, meets the target user's need for multiple source images, and improves the user experience.

[0103] S203: Perform image fusion on the attitude torsion image and the initial image to obtain a fused image.

[0104] Based on step S202, the target object in the initial image is subjected to pose torsion processing, so that the target object in the pose-torsion image has initially presented the target pose. However, the pose-torsion image obtained solely by pose torsion processing has low resolution, and the sharpness of the target object in the pose-torsion image is low. Figure 5 The target object in the shown pose torsion image lacks texture details, human body edges, and other information. Therefore, after obtaining the pose torsion image containing the target object with the target pose based on step S202, this embodiment of the application also supports enriching the details of the target object in the pose torsion image to obtain a fused image with higher clarity than the pose torsion image; wherein, the pose of the target object in the obtained fused image is the same as the pose of the target object in the pose torsion image, and the clarity of the fused image is higher than that of the pose torsion image.

[0105] It should be understood that since the initial image contains all the details of the target object, the pose torsion image can be refined based on the initial image to obtain a higher-resolution fused image. In specific implementations, the pose torsion image and the initial image can be fused to obtain a fused image. Image fusion can be understood as: fusing the features of the target object in the initial image with the features of the target object in the pose torsion image to obtain fused features, and then reconstructing the image based on the fused features to obtain the fused image. An example of fusing the pose torsion image and the fused image to obtain a higher-resolution fused image can be found in [reference needed]. Figure 7 ,like Figure 7 As shown, the fused image contains richer details of the target object compared to the pose torsion image. The above implementation method can enrich the details and texture of the target object, thereby improving the quality of the fused image.

[0106] S204: Extract a part image containing a specified part of the target object from the fused image, and perform image super-resolution processing on the part image to obtain the super-resolution part image.

[0107] S205: Fuse the super-resolution region image and the fused image to obtain the target image.

[0108] In steps S204-S205, the part image is an image containing only a specified part of the target object. This embodiment supports image super-resolution processing of the part image to meet the clarity requirements of different specified parts in different scenarios. For example, if the specified part is the face of the target object, then the part image can refer to an image containing the face of the target object. After performing image super-resolution processing on the part image, this embodiment can convert a low-resolution part image into a high-resolution part image, and the clarity of the super-resolution part image is higher than that of the part image before super-resolution processing. An exemplary schematic diagram of performing image super-resolution processing on a face image to obtain a super-resolution part image can be found [link to schematic diagram]. Figure 8 ,like Figure 8 As shown, image super-resolution processing is performed on the face image 801 extracted from the fused image and the face image 802 extracted from the initial image to obtain the super-resolution part image 803.

[0109] This application embodiment also supports fusing the super-resolution part image with the fused image to obtain a target image. The target image contains a target object that exhibits the target pose, and the specified part of the target object has high resolution, resulting in a high-quality target image. An exemplary schematic diagram of fusing a super-resolution face image and the fused image to obtain a target image can be found here. Figure 9 .

[0110] In this embodiment, the target object in the acquired initial image can be subjected to pose torsion processing, so that the distorted pose image contains the target object and the target object presents the target pose. Then, the initial image with richer details is used to fuse the pose torsion image with coarser details, resulting in a fused image with richer details. This achieves refinement of the target object presenting the target pose contained in the pose torsion image. Furthermore, since the target object contained in the pose torsion image already presents the target pose, the image fusion process can focus on enriching the detailed features of the pose torsion image without considering the pose torsion of the target object, reducing the difficulty of image fusion. In addition, it also supports image super-resolution processing on the part image containing a specified part of the target object extracted from the fused image, so that the specified part contained in the super-resolution part image has higher clarity, and image super-resolution processing of the part image containing the specified part can save image processing resources. Then, the super-resolution part image is fused with the fused image to obtain a target image with higher clarity.

[0111] Figure 10 This illustration shows a flowchart of an image processing method provided by an exemplary embodiment of this application. The embodiment is described using the example of the image processing method being executed by the aforementioned target terminal. This image processing method may include, but is not limited to, steps S1001-S1006:

[0112] S1001: Obtain the initial image containing the target object.

[0113] S1002: Acquire source image, which includes source object and the source object is in the target pose.

[0114] It should be noted that the specific implementation process shown in steps S1001-S1002 can be found in the relevant description of the specific implementation process shown in steps S201-S202 above, and will not be repeated here.

[0115] S1003: Perform pose torsion processing on the target object in the initial image based on the source image to obtain a pose torsion image.

[0116] If the source image contains a source object and the source object is in the target pose, then performing pose torsion processing on the target object in the initial image based on the source image can torsion the target object from the initial pose to the target pose. In a specific implementation, the steps of performing pose torsion processing on the target object in the initial image based on the source image to obtain the pose-torsion image may include, but are not limited to, steps s11-s13, wherein:

[0117] s11: Model the target object in the initial image to obtain the first model; and model the material object in the source image to obtain the second model. The model obtained by modeling any object includes: multiple meshes constituting the pose presented by the object, each mesh being associated with the texture information of the object; that is, by modeling any object, it is possible to divide the object into multiple facets using meshes, with one facet corresponding to one mesh. Each mesh is a polygon (such as a triangle, quadrilateral, or other simple convex polygon) composed of at least three mesh vertices, and each mesh vertex constituting the mesh may include mesh information (such as coordinate values, laser reflection intensity, color information, etc.); connecting multiple meshes in a certain order yields the model after modeling the object.

[0118] In specific implementation, this application embodiment supports the use of three-dimensional (3D) modeling technology to model the target object in the initial image and the material object in the source image, respectively. Thus, the first model corresponding to the target object and the second model corresponding to the source object are both 3D models (or three-dimensional models). This method of modeling the target object or source object using 3D modeling technology can obtain relatively rich three-dimensional data for any object. Subsequent image processing based on this rich 3D data can make the pose of the target object after twisting more closely match the target pose of the source object. The 3D modeling technology may include, but is not limited to: building blocks (e.g., dividing an object into many small pieces and then combining them to achieve modeling), mesh editing, skinning, and parametric multi-person linear model (SMPL) techniques. SMPL is a method for modeling the human body, which can perform arbitrary human body modeling and animation driving, and can simulate the bulges and depressions of human muscles during limb movement to a certain extent. Therefore, it can avoid surface distortion of the human body during movement and accurately depict the shape of muscle stretching and contraction movements. This application does not limit the specific 3D modeling technology used in its embodiments, but this will not be explained here.

[0119] An exemplary schematic diagram illustrating how to model a target object in an initial image to obtain a first model corresponding to the target object, and how to model a source object in a source image to obtain a second model corresponding to the source object, can be found here. Figure 11 ;like Figure 11As shown, after using 3D modeling technology (such as SMPL) to perform 3D modeling on the target object 301 in the initial image 101, a first model 1101 of the target object 301 (i.e., the 3D model of the target object) can be obtained; and after using 3D modeling technology to perform 3D modeling on the material object 501 in the material image 102, a second model 1102 of the material object 501 (i.e., the 3D model of the material object) can be obtained. The model obtained after 3D modeling any object includes multiple meshes; it should be noted that... Figure 11 The example only shows the mesh contained in the head of any model and does not limit the embodiments of this application. The embodiments of this application do not limit the number and shape of the mesh contained in any model.

[0120] This application embodiment also assigns numbers to all the meshes that make up any model, obtaining the number of each mesh in the multiple meshes included in the first model, and the number of each mesh in the multiple meshes included in the second model. For example, if the first model contains 10,000 meshes, then the 10,000 meshes can be numbered according to a numbering rule to obtain the number of each mesh; the numbering rule may include: numbering each mesh according to the structure of the human body from head to toe and from left to right, such as... Figure 11 As shown, the first grid 1103 on the top of the first model's head is numbered as grid 1, the grid 1104 adjacent to the left of the first grid 1103 is numbered as grid 2, ..., and the last grid of the toes is numbered as grid 10000. Similarly, assuming the second model contains 9800 grids, the grids are numbered in the same way to obtain the number of each grid. For example, the first grid 1105 on the top of the first model's head is numbered as grid 1, the grid 1106 adjacent to the left of the first grid 1105 is numbered as grid 2, ..., and the last grid of the toes is numbered as grid 9800. Of course, the above is only an exemplary way of numbering grids. In actual applications, the numbering rules can be changed, and this application embodiment does not limit the numbering rules.

[0121] s12: Based on the mesh correspondence between the first model and the second model, the texture information associated with each mesh in the first model is mapped to the corresponding mesh in the second model.

[0122] It should be noted that the meshes constituting any object's model are associated with the texture information of that object, which can also be simply referred to as texture. Texture information refers to the texture of any object's surface; texture information may include, but is not limited to, grooves that make the surface of any object appear uneven, or colored patterns that appear on the surface of any object; when texture information is mapped onto the model of any object, the model can appear more realistic. This application's embodiments support the use of an image affine transformation algorithm (warp algorithm) to map the texture information associated with each mesh in the first model to the corresponding meshes in the second model, so that the surface texture and color of the mapped second model are consistent with the surface texture and color of the target object in the initial image.

[0123] Among them, the image affine transformation algorithm is a method of deforming geometric shapes by scaling, distorting, or performing other operations on the geometric shapes. Specifically, based on a preset matrix, the texture information associated with each grid in the first model is mapped to each grid in the second model, so that the appearance of the target object (including clothing style, texture, color, etc.) in the target pose is consistent with the appearance of the target object in the initial image. The preset matrix is ​​determined based on the first and second models, and in this embodiment, it is referred to as the grid correspondence. That is, the texture mapping process described above can be realized through the grid correspondence between the first and second models. The so-called grid correspondence between the first and second models refers to the correspondence between the grids contained in the first model and the grids contained in the second model; as... Figure 11 As described in the illustrated embodiment, each grid in the first model is numbered, and each grid in the second model is numbered. Therefore, it is said that two grids in the first model with the same number as those in the second model have a correspondence. For example, if the grid numbers in the first model include: grid 1, grid 2, grid 3, ..., grid 10000, and the grid numbers in the second model include: grid 1, grid 2, grid 3, ..., grid 9800, then it is said that the grid numbered grid 1 in the first model corresponds to the grid numbered grid 1 in the second model.

[0124] The following is combined Figure 12 The diagram shown illustrates the process of texture mapping based on the mesh correspondence between the first and second models; for example... Figure 12As shown, after mapping the texture information associated with the grid numbered X (X is a positive integer and X is less than the total number of grids contained in the first model) in the first model 1101 to the grid numbered X in the second model 1102, the display area occupied by grid X in the second model 1102 presents the texture and color contained in the display area occupied by grid X in the first model 1101. After continuing to map the texture information associated with the grid in the first model 1101 to the corresponding grid in the second model 1102 according to the grid correspondence, the mapped second model 1201 can be obtained. The mapped second model 1201 presents the target pose, and the texture and color of any part matches the texture and color of the part corresponding to that part in the first model 1101. This makes the mapped second model 1201 look like the target object contained in the initial image, only the target object pose has changed from the initial pose presented in the initial image to the target pose.

[0125] It should be noted that when mapping the texture information associated with mesh X in the first model to mesh X in the second model, the shapes of mesh X in the first model and mesh X in the second model may not be the same; for example, mesh X in the first model is a right triangle, while mesh X in the second model is an equilateral triangle. If the texture information associated with mesh X in the first model is simply pasted onto mesh X in the second model, mesh X may appear less smooth and natural in the second model. Therefore, this embodiment also supports distorting the shape of mesh X in the first model according to the shape of mesh X in the second model during the process of pasting the texture information associated with mesh X in the first model onto mesh X in the second model, so that the distorted shape of mesh X in the first model is consistent with the shape of mesh X in the second model; specifically, the aforementioned warp algorithm can be used to distort mesh X in the first model.

[0126] s13: Using the mapped second model, generate an attitude torsion image.

[0127] Based on the above steps s11-s12, the mapped second model can be obtained. Then, an attitude torsion image can be generated based on the mapped second model. This attitude torsion image only contains the mapped second model, and the background of the attitude torsion image is a solid color (such as black or white).

[0128] In summary, after obtaining the initial image, the target object in the initial image can be subjected to pose torsion processing to obtain a pose torsion image. The target object in the pose torsion image presents the target pose. This method of preprocessing the pose of the target object in the initial image to obtain the target object presenting the target pose can avoid having to perform pose torsion on the target object when enhancing the details of the target object later, which can reduce the difficulty of enriching the details of the target object later.

[0129] S1004: Perform image fusion on the attitude torsion image and the initial image to obtain the fused image.

[0130] Step S1003 yields a pose torsion image of the target object presenting the target pose. However, relying solely on mapping the texture information of the first model corresponding to the target object to the second model corresponding to the material object to generate the pose torsion image results in missing edges (or contours) and lost texture details of the target object in the pose torsion image. Consequently, although the target object presents the target pose in the pose torsion image, the resolution of the pose torsion image is low and the details are not clear enough. Therefore, this embodiment also supports detail enhancement of the pose torsion image to obtain a higher-resolution fused image. Considering that the initial image is a high-resolution image uploaded by the target user, this embodiment can fuse the high-resolution initial image with the pose torsion image to obtain a higher-resolution fused image.

[0131] In specific implementations, this application embodiment supports the use of Generative Adversarial Networks (GANs) to perform the image fusion process described above for the pose twisting image and the initial image; GAN networks are a type of deep learning model in Artificial Intelligence (AI) technology. Artificial intelligence utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results—theories, methods, technologies, and application systems. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making functions. Artificial intelligence technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The GAN networks mentioned above involve machine learning / deep learning and other areas within the field of artificial intelligence.

[0132] A GAN network can include at least two modules: a generative model and a discriminative model. These modules learn from each other through a game-like interaction to produce better output results. Taking images as input data for a GAN network, and assuming the GAN network can generate images containing cats, we will briefly introduce the generative and discriminative modules. The generative module processes one or more frames of input images containing cats to generate a new image containing a cat that is not present in the original input frames. The discriminative module judges the input image to determine whether the animal in the image is a cat. During the training of the GAN network, the images generated by the generative module are fed to the discriminative module for judgment. Based on the judgment results, the parameters of the GAN network are continuously adjusted until the trained GAN network can accurately generate new images and accurately discriminate between them. It should be noted that, depending on the training objectives of the GAN network, the modules included in the GAN network are not limited to the two types of modules mentioned above.

[0133] To better understand the process of using a GAN network to perform image fusion on a pose twisting image and an initial image to obtain a fused image, the following section combines the attached diagram. Figure 13a The GAN generator network structure diagram shown illustrates the specific implementation of image fusion. This GAN generator network structure diagram corresponds to the network structure diagram of the generator module included in the aforementioned GAN network. Figure 13a As shown, the GAN generative network structure includes: an autoencoder (AE), a backbone transfer network, and a feature fusion module; where: ① An autoencoder is a neural network architecture that includes an encoding architecture (or encoder) and a decoding architecture (or decoder); the encoding architecture consists of a series of hidden layers with progressively decreasing dimensions. The encoding architecture can reduce the dimensionality of input data (such as an initial image) to obtain a lower-dimensional information representation (such as image features); the decoding architecture consists of a series of hidden layers with progressively increasing dimensions. Its grid structure can be understood as a mirror image of the encoding architecture. The decoding architecture can reconstruct the low-dimensional information representation output by the encoding architecture to obtain a higher-dimensional output result. The dimension of the hidden layers mentioned above can be determined by the number of neurons contained in the hidden layer. For example, if hidden layer 1 contains 2000 neurons and hidden layer 2 contains 1000 neurons, then the dimension of hidden layer 1 is higher than that of hidden layer 2. ② A backbone transfer network is a neural network architecture. Similar to an autoencoder, it also includes an encoding architecture and a decoding architecture. For details on the encoding and decoding architectures, please refer to the aforementioned description of autoencoders; they will not be repeated here. ③ The feature fusion module includes multiple sub-feature fusion modules (such as...). Figure 13a The spade module shown can fuse two features extracted from the initial image and the pose torsion image through each sub-feature fusion module to obtain the fused feature of the two features.

[0134] The above provides a brief overview of the composition of the GAN generative network structure. The implementation of fusing a pose twisting image and an initial image using a GAN generative network to obtain a fused image can include: First, performing feature extraction on the target object in the initial image to obtain a first feature map; and then performing feature extraction on the pose twisting image to obtain a second feature map; wherein each feature map includes multiple features; second, according to the feature correspondence between the first and second feature maps, replacing the corresponding features in the second feature map with features from the first feature map; finally, performing image reconstruction based on the feature-replaced second feature map to obtain the fused image. Further combining... Figure 13a The GAN generator network structure shown describes the detailed process of the above image fusion, which may include, but is not limited to, steps s21-s24. Wherein:

[0135] s21: The initial image, which includes the target object, is used as input data to the autoencoder in the GAN generator network structure; and the pose torsion image is used as input data to the backbone transfer network in the GAN generator network structure.

[0136] s22: An autoencoder is used to extract features from the target object in the initial image, resulting in a first feature map. This first feature map includes multiple features of the target object; for example, the first feature map contains first feature 1, first feature 2, first feature 3, ..., first feature M, where M is an integer greater than 3. First feature 1 of the target object can be extracted through the first convolutional layer of the autoencoder; then, first feature 1 is extracted through the second convolutional layer of the autoencoder to obtain first feature 2; the third convolutional layer of the autoencoder is then used to extract first feature 2 to obtain first feature 3, and so on, until first feature M is extracted through the last convolutional layer of the autoencoder. Similarly, a backbone transfer network is used to extract features from the target object in the pose torsion image, resulting in a second feature map. This second feature map contains second feature 1 extracted through the first convolutional layer of the backbone transfer network, second feature 2 extracted through the second convolutional layer, and so on.

[0137] s23: Based on the feature correspondence between the first feature map (including first feature 1) and the second feature map (including second feature 1), and using the various sub-feature fusion modules in the feature fusion module to fuse the first feature map and the second feature map, the fusion process here may include replacing the corresponding features in the second feature map with the features in the first feature map, such as replacing the second feature 1 in the second feature map with the first feature 1 in the first feature map. Specifically, the feature fusion process may include: using the first feature 1 extracted from the initial image by the first convolutional layer in the autoencoder and the second feature 1 extracted from the pose torsion image by the first convolutional layer in the backbone transfer network as input data for the first sub-feature fusion module in the feature fusion module; then using the first fused feature output by the first sub-feature fusion module and the second feature 1 output by the first convolutional layer in the backbone transfer network as input data for the second convolutional layer in the backbone transfer network, and using the output of the second convolutional layer as the second feature 2 of the target object. The second sub-feature fusion module in the feature fusion module is then used to fuse the first feature 2 and the second feature 2 to obtain the second fused feature. The second fused feature and second feature 2 are then used as input data for the third convolutional layer in the backbone transfer network, and the output of the third convolutional layer is used as the second feature 3 of the target object. The above process is repeated until the second feature M output by the last convolutional layer in the backbone transfer network and the Mth fused feature output by the last sub-feature fusion module in the feature fusion module are obtained.

[0138] s24: The second feature map after feature replacement can be obtained through step s23. The second feature map contains: the second feature M output by the last convolutional layer in the backbone transfer network, and the Mth fused feature output by the last sub-feature fusion module in the feature fusion module. The process of image reconstruction based on the second feature map after feature replacement may include: reconstructing a color mode image containing the target object presenting the target pose, and a mask image containing the target object presenting the target pose, based on the second feature map after feature replacement; generating a fused image based on the color mode image and the mask image.

[0139] The image reconstruction process described above essentially involves using an upsampling network (such as a neural network with upsampling capabilities) to reconstruct the second feature map after feature replacement. This means generating an image based on the features contained in the second feature map after feature replacement. This image includes a color mode image and a mask image. The color mode image is an RGB (color system) image containing the target object representing the target pose. This RGB image contains three color channels: red (R), green (G), and blue (B), giving the RGB image a color mode. The mask image is an alpha image associated with the color mode image. This association means that the mask image also includes the target object representing the target pose, as contained in the color mode image. This alpha image contains an alpha channel, with values ​​between 0 and 1. When the alpha channel value is 0, the corresponding pixel is initialized to 0, and the image composed of that pixel is displayed as black. When the alpha channel value is 1, the corresponding pixel value is set to 255, and the image composed of that pixel is displayed as white. Adjusting the alpha channel value adjusts the image's transparency.

[0140] Based on the above description, an exemplary schematic diagram illustrating the generation of a fused image from a color mode image containing a target object representing a target pose and a mask image can be found here. Figure 13b ,like Figure 13b As shown, the pixel values ​​of the display area occupied by the target object in the mask image 1301 are set to 0, that is, the target object appears black, while the pixel values ​​of the display area occupied by the target object are set to 255. Thus, the display area occupied by the target object in the mask image 1301 is black, and the other areas are displayed as white. By multiplying the mask image 1301 by the color mode image 1302, an intermediate image 1303 containing the target object presenting the target pose can be obtained. In this intermediate image 1303, the display area occupied by the target object is opaque, but the other areas are transparent.

[0141] It should be noted that the initial image often includes not only the target object but also the image background. The image background refers to the content in the initial image other than the target object. Therefore, to achieve the feature extraction of the target object from the initial image mentioned in step s22, this embodiment supports performing foreground segmentation on the initial image before executing step s22, obtaining a foreground image and a background image containing the target object, and then performing feature extraction on the foreground image to obtain a first feature map. This feature extraction processing based on the foreground image can obtain more accurate features of the target object, thereby improving the quality of the fused image after subsequent feature fusion. A schematic diagram of an interface for performing foreground segmentation on an image (such as any image) to obtain a background image and a foreground image can be found here. Figure 14 ,like Figure 14 As shown, image 1401 includes foreground image 1402 and background image 1403. Foreground image 1402 includes only the target object, while background image 1403 includes other content in the initial image besides the target object. For example, background image 1403 includes a street view.

[0142] Through steps s21-s24 above, the pose torsion image is refined using the initial image to obtain a high-resolution intermediate image containing the target object. When no background needs to be added to the target object, the intermediate image can be designated as the fused image; when a background needs to be added to the target object, the image with the added background can be designated as the fused image. For example, adding a sports field background to the target object indicates that the target object is in a sports field scene. By adding a background to the target object, the target object presenting the target pose can be added to any scene, increasing the fun of human pose transfer and meeting the target user's scene requirements for transferring the target object to different scenes.

[0143] This application embodiment also supports adding a background to the target object. In this implementation, the fused image includes: the target object presenting the target pose and the background. In a specific implementation, the process of generating a fused image containing background content may include: first, obtaining a target background image, which may include: the background image of an initial image, or the background image of a source image, or a custom background image (such as one uploaded or selected by the user); second, performing image fusion on the color mode image and the mask image to obtain an intermediate image; finally, superimposing the target background image onto the intermediate image to obtain a fused image, which contains the background content contained in the target background image.

[0144] In one implementation, the target background image is the background image of the source image. For example... Figure 15As shown, after foreground segmentation processing of the source image 102, a foreground image 1501 and a background image 1502 of the source image 102 can be obtained; then, the background image 1502 is superimposed on the intermediate image 1503 to obtain a fused image 1504. Through the above process, the pose of the target object in the initial image can be transferred to the target pose of the source object, and the scene of the target object in the initial image can be transferred to the scene where the source object is located in the source image. It should be noted that after foreground segmentation processing of the source image (or the initial image), the target object is removed from the background image of the obtained source image, resulting in the display area occupied by the target object in the background image being displayed as black (or white), that is, the pixel value corresponding to the display area occupied by the target object is 0 (or 1); if the intermediate image is superimposed on the background image, the resulting image will not be natural. Therefore, this embodiment also supports filling processing of the background image with the target object removed, so that the display area occupied by the target object in the filled background image presents a color similar to the area adjacent to the display area in the background image. Optionally, the pixels in the display area occupied by the target object can be interpolated using the pixel values ​​surrounding the display area occupied by the target object, making the color of the display area occupied by the target object appear more natural with its surroundings after interpolation. Optionally, a neural network model with interpolation function can be used to interpolate the background image from which the target object has been removed. This application does not limit the neural network model with interpolation function, but it is described here.

[0145] In other implementations, the target background image can also be a user-defined background image. Implementations where the target user customizes the target background image can include: displaying a data recommendation interface, which includes one or more multimedia data; when any multimedia data in the data recommendation interface is selected, segmenting the target background image from the selected multimedia data; wherein the selected multimedia data can be an image or a video stream, and when the selected multimedia data is a video stream, the target background image is segmented from any image frame in the selected multimedia data. When the target background image is a user-defined background image, it is not necessary to perform the foreground segmentation processing described above on the background image; instead, the background image can be directly superimposed onto the intermediate image. An exemplary flowchart of the interface for obtaining an intermediate image by fusing a color mode image and a mask image, and superimposing the target background image onto the intermediate image to obtain the fused image, can be found in [reference needed]. Figure 16 ,like Figure 16As shown, the target background image is a user-defined image. First, by multiplying the mask image 1301 by the color mode image 1302, an intermediate image containing the target object presenting the target pose can be obtained. Then, by combining the intermediate image with the target background image 1601, a fused image 1504 can be obtained.

[0146] It is worth noting that the embodiments of this application also support superimposing multiple intermediate images, such as those containing target objects in different target poses, onto a single target background image. In this implementation, multiple target objects presenting different target poses can be displayed in a single fused image. Furthermore, if multiple target videos containing target objects are generated, the same background can be added to the multiple target videos, and the multiple videos can be fused to obtain a new target video. When playing this new target video, any video frame can include multiple target objects, and these multiple target objects can present the same or different target poses.

[0147] S1005: Extract a part image containing a specified part of the target object from the fused image, and perform image super-resolution processing on the part image to obtain the super-resolution part image.

[0148] S1006: Fuse the super-resolution part image and the fused image to obtain the target image.

[0149] In steps S1005-S1006, as described above, since the initial image has a higher resolution and higher clarity, this embodiment supports extracting an image containing a specified part of the target object from the initial image as a reference image; and performing image super-resolution processing based on the reference image and the part image to obtain the super-resolution part image. Specifically, an image containing a specified part of the target object can be extracted from the initial image as a reference image, the reference image having higher clarity than the part image; feature extraction is performed on the reference image to obtain reference image features, and feature extraction is performed on the part image to obtain part image features; then the reference image features and part image features are fused, and image reconstruction is performed based on the fused features to obtain the super-resolution part image.

[0150] Specifically, a region enhancement network can be used to perform the image super-resolution processing described above. The type of region enhancement network varies depending on the specific region indicated by the cropped region image. This embodiment uses a face as an example to describe the process of performing image super-resolution processing using a face enhancement network. A schematic diagram of the face enhancement network architecture can be found in [reference needed]. Figure 17 ,like Figure 17As shown, this face enhancement network includes at least a reconstruction module and a face identification module. The reconstruction module can reconstruct the input data, such as an input part image, and reconstruct the part image to obtain the reconstructed part image. The face identification module can be used to detect whether the face contained in the reconstructed part image is consistent with the face of the target object contained in the reference image. If they are consistent, the reconstruction is considered successful; if they are inconsistent, the reconstruction is considered a failure, and the parameters of the face enhancement network are adjusted to obtain a face enhancement network that can correctly reconstruct faces. It should be noted that... Figure 17 The face enhancement network mentioned can be a GAN network with face enhancement function. This application does not limit the specific structure and type of the face enhancement network.

[0151] The clarity of the super-resolution region image obtained based on the above implementation method is higher than that of the specified region in the fused image. Therefore, by fusing the super-resolution region image with the fused image, such as pasting the super-resolution region image into the specified region of the fused image, a target image with high clarity of the specified region can be obtained. This application embodiment supports detail enhancement of specified regions (such as faces) in the fused image, meeting the target user's detail requirements for the specified region, significantly improving the clarity of the texture of the specified region, and also enhancing the recognizability of the target image.

[0152] In this embodiment, the target object in the acquired initial image can be subjected to pose torsion processing, so that the distorted pose image contains the target object and the target object presents the target pose. Then, the initial image with richer details is used to fuse the pose torsion image with coarser details, resulting in a fused image with richer details. This achieves refinement of the target object presenting the target pose contained in the pose torsion image. Furthermore, since the target object contained in the pose torsion image already presents the target pose, the image fusion process can focus on enriching the detailed features of the pose torsion image without considering the pose torsion of the target object, reducing the difficulty of image fusion. In addition, it also supports image super-resolution processing on the part image containing a specified part of the target object extracted from the fused image, so that the specified part contained in the super-resolution part image has higher clarity, and image super-resolution processing of the part image containing the specified part can save image processing resources. Then, the super-resolution part image is fused with the fused image to obtain a target image with higher clarity.

[0153] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0154] Figure 18This application shows a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment; the image processing apparatus can be used to perform... Figure 2 or Figure 10 Some or all of the steps in the method embodiments shown. Please refer to [link / reference]. Figure 18 The image processing apparatus includes the following units:

[0155] The acquisition unit 1801 is used to acquire an initial image containing the target object and perform pose torsion processing on the target object in the initial image to obtain a pose torsion image.

[0156] Processing unit 1802 is used to perform image fusion on the posture torsion image and the initial image to obtain a fused image; the posture of the target object in the fused image is the same as the posture of the target object in the posture torsion image.

[0157] The acquisition unit 1801 is used to extract a part image containing a specified part of the target object from the fused image, and perform image super-resolution processing on the part image to obtain the super-resolution part image.

[0158] The processing unit 1802 is used to fuse the super-resolution part image and the fused image to obtain the target image.

[0159] In one implementation, the target object in the initial image presents an initial pose. The processing unit 1802 is used to perform pose torsion processing on the target object in the initial image to obtain a pose torsion image. Specifically, it is used for:

[0160] Acquire source images, which include source objects and the source objects are presented in the target pose;

[0161] The target object in the initial image is modeled to obtain the first model; the material object in the source image is modeled to obtain the second model; the model obtained by modeling any object includes: multiple meshes constituting the pose presented by any object, and each mesh is associated with the texture information of any object;

[0162] Based on the mesh correspondence between the first model and the second model, the texture information associated with each mesh in the first model is mapped to the corresponding mesh in the second model;

[0163] The mapped second model is used to generate an attitude torsion image.

[0164] In one implementation, when the processing unit 1802 acquires the source image, it is specifically used for:

[0165] Obtain the source video, which consists of N image frames, where N is a positive integer.

[0166] Iterate through each image frame in the source video and determine the currently iterated image frame as the source image.

[0167] In one implementation, the processing unit 1802 is further configured to:

[0168] After all the image frames in the source video have been traversed, N target images are obtained, and one target image corresponds to one image frame in the source video.

[0169] The N target images are arranged in order according to the playback order of the image frames corresponding to each target image in the source video to obtain the target video.

[0170] In one implementation, the processing unit 1802 is used to perform image fusion on the attitude torsion image and the initial image, and when obtaining the fused image, it is specifically used for:

[0171] The target object in the initial image is subjected to feature extraction processing to obtain a first feature map; and the pose torsion image is subjected to feature extraction processing to obtain a second feature map; wherein each feature map includes multiple features;

[0172] Based on the feature correspondence between the first feature map and the second feature map, each feature in the first feature map is used to replace the corresponding feature in the second feature map;

[0173] Image reconstruction is performed based on the second feature map after feature replacement to obtain the fused image.

[0174] In one implementation, the processing unit 1802 is used to perform feature extraction processing on the target object in the initial image to obtain the first feature map, specifically for:

[0175] Perform foreground segmentation on the initial image to obtain a foreground image containing the target object;

[0176] Feature extraction is performed on the foreground image to obtain the first feature map.

[0177] In one implementation, the processing unit 1802 is used to perform image reconstruction based on the second feature map after feature replacement, and to obtain a fused image, specifically for:

[0178] Based on the second feature map after feature replacement, a color mode image containing the target object presenting the target pose and a mask image containing the target object presenting the target pose are reconstructed.

[0179] A fused image is generated based on the color mode image and the mask image.

[0180] In one implementation, when the processing unit 1802 generates the fused image based on the color mode image and the mask image, it specifically performs the following:

[0181] Obtain the target background image, which may include: the background image of the initial image, the background image of the source image, or a custom background image;

[0182] The color mode image and the mask image are fused to obtain an intermediate image;

[0183] The target background image is superimposed onto the intermediate image to obtain the fused image.

[0184] In one implementation, when the target background image is a custom background image, the processing unit 1802 is further configured to:

[0185] Display a data recommendation interface, which includes one or more multimedia data.

[0186] When any multimedia data in the data recommendation interface is selected, the target background image is segmented from the selected multimedia data.

[0187] The selected multimedia data is either an image or a video stream. When the selected multimedia data is a video stream, the target background image is segmented from any image frame in the selected multimedia data.

[0188] In one implementation, the processing unit 1802 is used to perform image super-resolution processing on the part image, and when obtaining the super-resolution part image, it is specifically used for:

[0189] An image containing a specified part of the target object is extracted from the initial image and used as a reference image; wherein, the reference image has higher resolution than the image of the part.

[0190] Feature extraction is performed on the reference image to obtain reference image features, and feature extraction is performed on the part image to obtain part image features;

[0191] The features of the reference image and the feature image are fused, and the image is reconstructed based on the fused features to obtain the super-resolution feature image.

[0192] According to one embodiment of this application, Figure 18The various units in the image processing apparatus shown can be individually or entirely combined into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This can achieve the same operation without affecting the technical effects of the embodiments of this application. The above units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the image processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by multiple units working together. According to another embodiment of this application, the image processing apparatus can be executed by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access storage medium (RAM), and read-only storage medium (ROM). Figure 2 and Figure 10 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 18 The image processing apparatus shown herein, and the image processing method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and executed therein.

[0193] In this embodiment, the processing unit 1802 can perform pose torsion processing on the target object in the acquired initial image, so that the torsion image contains the target object and the target object presents the target pose. The processing unit 1802 also uses the initial image with richer details to perform image fusion on the pose torsion image with coarser details, resulting in a fused image with richer details. This achieves refinement of the target object presenting the target pose contained in the pose torsion image. Furthermore, since the target object contained in the pose torsion image already presents the target pose, the image fusion process can focus on enriching the detailed features of the pose torsion image without considering the pose torsion of the target object, reducing the difficulty of image fusion. In addition, it also supports image super-resolution processing on the part image containing a specified part of the target object extracted from the fused image, so that the specified part contained in the super-resolution part image has higher clarity. Performing image super-resolution processing on the part image containing the specified part can save image processing resources. Then, the super-resolution part image is fused with the fused image to obtain a target image with higher clarity.

[0194] Figure 19 A schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of this application is shown. Please refer to... Figure 19The image processing device includes a processor 1901, a communication interface 1902, and a computer-readable storage medium 1903. The processor 1901, communication interface 1902, and computer-readable storage medium 1903 can be connected via a bus or other means. The communication interface 1902 is used to receive and send data. The computer-readable storage medium 1903 can be stored in the memory of the image processing device. The computer-readable storage medium 1903 stores computer programs, including program instructions. The processor 1901 executes the program instructions stored in the computer-readable storage medium 1903. The processor 1901 (or CPU (Central Processing Unit)) is the computing and control core of the image processing device, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve corresponding method flows or corresponding functions.

[0195] This application embodiment also provides a computer-readable storage medium (Memory), which is a memory device in an image processing apparatus for storing programs and data. It is understood that the computer-readable storage medium here may include both the built-in storage medium in the image processing apparatus and extended storage media supported by the image processing apparatus. The computer-readable storage medium provides storage space that stores the processing system of the image processing apparatus. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by the processor 1901, which may be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here may be high-speed RAM memory or non-volatile memory, such as at least one disk storage device; optionally, it may also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0196] In one embodiment, the image processing device may be the target terminal or server mentioned in the foregoing embodiments; the computer-readable storage medium stores one or more instructions; the processor 1901 loads and executes one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above-described display processing method embodiments; specifically, the one or more instructions in the computer-readable storage medium are loaded by the processor 1901 and executed as follows:

[0197] Obtain an initial image containing the target object, and perform pose torsion processing on the target object in the initial image to obtain a pose torsion image;

[0198] The pose torsion image and the initial image are fused to obtain a fused image; the pose of the target object in the fused image is the same as the pose of the target object in the pose torsion image.

[0199] Extract the part image containing the specified part of the target object from the fused image, and perform image super-resolution processing on the part image to obtain the super-resolution part image;

[0200] The target image is obtained by fusing the super-resolution region image and the fused image.

[0201] In one implementation, the target object in the initial image is in an initial pose. One or more instructions in the computer-readable storage medium are loaded by the processor 1901 and executed to perform pose torsion processing on the target object in the initial image to obtain a pose-torsion image. Specifically, the following steps are performed:

[0202] Acquire source images, which include source objects and the source objects are presented in the target pose;

[0203] The target object in the initial image is modeled to obtain the first model; the material object in the source image is modeled to obtain the second model; the model obtained by modeling any object includes: multiple meshes constituting the pose presented by any object, and each mesh is associated with the texture information of any object;

[0204] Based on the mesh correspondence between the first model and the second model, the texture information associated with each mesh in the first model is mapped to the corresponding mesh in the second model;

[0205] The mapped second model is used to generate an attitude torsion image.

[0206] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 1901 and, when acquiring the source image, specifically perform the following steps:

[0207] Obtain the source video, which consists of N image frames, where N is a positive integer.

[0208] Iterate through each image frame in the source video and determine the currently iterated image frame as the source image.

[0209] In one implementation, one or more instructions in a computer-readable storage medium are loaded by processor 1901 and further executed as follows:

[0210] After all the image frames in the source video have been traversed, N target images are obtained, and one target image corresponds to one image frame in the source video.

[0211] The N target images are arranged in order according to the playback order of the image frames corresponding to each target image in the source video to obtain the target video.

[0212] In one implementation, when one or more instructions in the computer-readable storage medium are loaded by the processor 1901 and executed to fuse the attitude torsion image and the initial image to obtain the fused image, the following steps are specifically performed:

[0213] The target object in the initial image is subjected to feature extraction processing to obtain a first feature map; and the pose torsion image is subjected to feature extraction processing to obtain a second feature map; wherein each feature map includes multiple features;

[0214] Based on the feature correspondence between the first feature map and the second feature map, each feature in the first feature map is used to replace the corresponding feature in the second feature map;

[0215] Image reconstruction is performed based on the second feature map after feature replacement to obtain the fused image.

[0216] In one implementation, when one or more instructions in the computer-readable storage medium are loaded by the processor 1901 and executed to perform feature extraction processing on the target object in the initial image to obtain the first feature map, the following steps are specifically performed:

[0217] Perform foreground segmentation on the initial image to obtain a foreground image containing the target object;

[0218] Feature extraction is performed on the foreground image to obtain the first feature map.

[0219] In one implementation, when one or more instructions in the computer-readable storage medium are loaded by the processor 1901 and executed to reconstruct the image based on the second feature map after feature substitution to obtain the fused image, the following steps are specifically performed:

[0220] Based on the second feature map after feature replacement, a color mode image containing the target object presenting the target pose and a mask image containing the target object presenting the target pose are reconstructed.

[0221] A fused image is generated based on the color mode image and the mask image.

[0222] In one implementation, when one or more instructions in the computer-readable storage medium are loaded by the processor 1901 and executed to generate a fused image based on the color mode image and the mask image, the following steps are specifically performed:

[0223] Obtain the target background image, which may include: the background image of the initial image, the background image of the source image, or a custom background image;

[0224] The color mode image and the mask image are fused to obtain an intermediate image;

[0225] The target background image is superimposed onto the intermediate image to obtain the fused image.

[0226] In one implementation, when the target background image is a custom background image, one or more instructions in the computer-readable storage medium are loaded by the processor 1901 and the following steps are also executed:

[0227] Display a data recommendation interface, which includes one or more multimedia data.

[0228] When any multimedia data in the data recommendation interface is selected, the target background image is segmented from the selected multimedia data.

[0229] The selected multimedia data is either an image or a video stream. When the selected multimedia data is a video stream, the target background image is segmented from any image frame in the selected multimedia data.

[0230] In one implementation, when one or more instructions in the computer-readable storage medium are loaded by the processor 1901 and executed to perform image super-resolution processing on a part image to obtain a super-resolution part image, the following steps are specifically performed:

[0231] An image containing a specified part of the target object is extracted from the initial image and used as a reference image; wherein, the reference image has higher resolution than the image of the part.

[0232] Feature extraction is performed on the reference image to obtain reference image features, and feature extraction is performed on the part image to obtain part image features;

[0233] The features of the reference image and the feature image are fused, and the image is reconstructed based on the fused features to obtain the super-resolution feature image.

[0234] In this embodiment, the processor 1901 can perform pose torsion processing on the target object in the acquired initial image, so that the torsion image contains the target object and the target object presents the target pose. Then, the processor 1901 uses the initial image with richer details to perform image fusion on the pose torsion image with coarser details, resulting in a fused image with richer details. This achieves refinement of the target object presenting the target pose contained in the pose torsion image. Furthermore, since the target object contained in the pose torsion image already presents the target pose, the image fusion process can focus on enriching the detailed features of the pose torsion image without considering the pose torsion of the target object, reducing the difficulty of image fusion. In addition, it also supports image super-resolution processing on the part image containing a specified part of the target object extracted from the fused image, so that the specified part contained in the super-resolution part image has higher clarity. Moreover, performing image super-resolution processing on the part image containing the specified part can save image processing resources. Then, the super-resolution part image is fused with the fused image to obtain a target image with higher clarity.

[0235] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of an image processing device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the image processing device to perform the above-described display processing method.

[0236] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0237] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0238] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this invention should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, Including: Obtain an initial image including a target object, and perform pose twisting processing on the target object in the initial image to obtain a pose-twisted image; Perform feature extraction processing on the target object in the initial image to obtain a first feature map; and perform feature extraction processing on the pose-twisted image to obtain a second feature map; wherein, each feature map includes multiple features; According to the feature correspondence between the first feature map and the second feature map, use each feature in the first feature map to replace the corresponding feature in the second feature map; Perform image reconstruction based on the second feature map after feature replacement to obtain a fused image; the pose of the target object in the fused image is the same as the pose of the target object in the pose-twisted image, and the clarity of the fused image is higher than the clarity of the pose-twisted image; Extract a part image including a specified part of the target object from the fused image, and perform image super-resolution processing on the part image with reference to the image of the corresponding specified part of the target object in the initial image to obtain a super-resolved part image; wherein, the clarity of the reference image is higher than the clarity of the part image; Fuse the super-resolved part image and the fused image to obtain a target image.

2. The method according to claim 1, wherein The target object in the initial image presents an initial pose, and the performing pose twisting processing on the target object in the initial image to obtain a pose-twisted image includes: Obtain a material image, the material image includes a material object and the material object presents a target pose; Model the target object in the initial image to obtain a first model; model the material object in the material image to obtain a second model; the model obtained by modeling any object includes: multiple meshes constituting the pose presented by the any object, and each mesh is associated with the texture information of the any object; Based on the mesh correspondence between the first model and the second model, map the texture information associated with each mesh in the first model to the corresponding mesh in the second model; Generate a pose-twisted image using the second model after mapping.

3. The method according to claim 2, wherein The obtaining the material image includes: Obtain a material video, the material video includes N image frames, and N is an integer greater than zero; Traverse each image frame in the material video, and determine the currently traversed image frame as the material image.

4. The method according to claim 3, wherein The method further includes: When all image frames in the material video have been traversed, obtain N target images, and one target image corresponds to one image frame in the material video; Arrange the N target images in order according to the playing order of the image frames corresponding to the respective target images in the material video to obtain a target video.

5. The method according to claim 1, wherein The performing feature extraction processing on the target object in the initial image to obtain a first feature map includes: Perform foreground segmentation processing on the initial image to obtain a foreground image including the target object; Perform feature extraction processing on the foreground image to obtain a first feature map.

6. The method according to claim 1, wherein The performing image reconstruction based on the second feature map after feature replacement to obtain a fused image includes: Based on the second feature map after feature replacement, reconstruct a color mode image including the target object presenting the target pose, and a mask image including the target object presenting the target pose; Generate a fused image according to the color mode image and the mask image.

7. The method according to claim 6, wherein The generating a fused image according to the color mode image and the mask image includes: Obtain a target background image, where the target background image includes: the background image of the initial image, or the background image of the material image, or a custom background image; Perform image fusion on the color mode image and the mask image to obtain an intermediate image; Overlay the target background image onto the intermediate image to obtain a fused image.

8. The method according to claim 7, wherein When the target background image is a custom background image, the method further includes: Display a data recommendation interface, where the data recommendation interface includes one or more multimedia data; When any multimedia data in the data recommendation interface is selected, segment the target background image from the selected multimedia data; Wherein, the selected multimedia data is an image or a video stream, and when the selected multimedia data is a video stream, the target background image is segmented from any image frame in the selected multimedia data.

9. The method according to claim 1, wherein The performing image super-resolution processing on the part image with reference to the image of the corresponding specified part of the target object in the initial image to obtain a super-resolved part image includes: Extract the image including the specified part of the target object from the initial image as a reference image; Extract features from the reference image to obtain reference image features, and extract features from the part image to obtain part image features; Fuse the reference image features and the part image features, and perform image reconstruction based on the fused features to obtain a super-resolved part image.

10. An image processing apparatus, characterized in that, Includes: An acquisition unit for acquiring an initial image including a target object, and performing pose twisting processing on the target object in the initial image to obtain a pose-twisted image; A processing unit for performing feature extraction processing on the target object in the initial image to obtain a first feature map; and performing feature extraction processing on the pose-twisted image to obtain a second feature map; wherein, each feature map includes multiple features; according to the feature correspondence between the first feature map and the second feature map, use each feature in the first feature map to replace the corresponding feature in the second feature map; perform image reconstruction based on the second feature map after feature replacement to obtain a fused image; the pose of the target object in the fused image is the same as the pose of the target object in the pose-twisted image, and the clarity of the fused image is higher than the clarity of the pose-twisted image; The acquisition unit is further configured to extract a part image including the target object from the fused image, and perform image super-resolution processing on the part image with reference to the image of the corresponding specified part of the target object in the initial image to obtain a super-resolved part image; wherein, the clarity of the reference image is higher than the clarity of the part image; The processing unit is further configured to fuse the super-resolved partial image and the fused image to obtain a target image.

11. An image processing device, characterized in that a processor adapted to execute a computer program; a computer-readable storage medium storing a computer program, which when executed by the processor, implements the image processing method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is adapted to be loaded and executed by the processor to implement the image processing method according to any one of claims 1-9.

13. A computer program product, characterized in that, The computer program product includes computer instructions, which when executed by the processor, implement the image processing method according to any one of claims 1-9.

Images