Image processing method and device, computer device and storage medium

Abstract

The application relates to an image processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: determining an initial texture map of a first object image; each first object pixel point in the first object image has a texture coordinate, and in the initial texture map, the texture coordinate of each first object pixel point is stored with the pixel coordinate of the first object pixel point; there are blank texture coordinates in the initial texture map; performing texture completion on the initial texture map to obtain a target texture map; the target texture map is obtained by storing the corresponding pixel coordinates at the blank texture coordinates in the initial texture map; performing pose feature transformation on a first feature map based on the texture coordinates of each second object pixel point in a second object image and the target texture map to obtain a third feature map; and performing clothing transformation based on a second feature map and the third feature map to generate a transformed object image corresponding to the second object image. The method can improve the clothing transformation effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] Online shopping has become an indispensable form of shopping, and clothing is a major category sold on e-commerce platforms. Consumers can choose clothing based on images displayed on e-commerce platforms, but they cannot know how it will look when actually worn. With the development of computer vision, virtual try-on has emerged to solve this problem. Through virtual try-on technology, the target clothing selected by the user can be transformed into the corresponding position in the user's image to obtain the transformed image of the clothing, i.e., the try-on effect image.

[0003] In traditional techniques, the target clothing area in the model image is usually overlaid onto the target clothing area in the user image to obtain the image with the clothing transformed.

[0004] However, since the pose of the model in the model image is different from the pose of the user in the user image, the traditional method of overlaying the target clothing area in the model image onto the target clothing area in the user image results in an unrealistic image after clothing transformation, leading to poor clothing transformation effect. Summary of the Invention

[0005] Therefore, it is necessary to provide an image processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the effect of clothing transformation in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides an image processing method. The method includes: determining an initial texture map of a first object image; each first object pixel in the first object image has texture coordinates, and the initial texture map stores the pixel coordinates of the first object pixel at the texture coordinates of the first object pixel; the initial texture map contains blank texture coordinates; performing texture completion on the initial texture map to obtain a target texture map; the target texture map is obtained by storing the corresponding pixel coordinates at the blank texture coordinates in the initial texture map; performing pose feature transformation on a first feature map based on the texture coordinates of each second object pixel in a second object image and the target texture map to obtain a third feature map; the first feature map is obtained by feature extraction from the first object image; performing clothing transformation based on the second feature map and the third feature map to generate a transformed object image corresponding to the second object image; the second feature map is obtained by feature extraction from the second object image.

[0007] Secondly, this application also provides an image processing apparatus. The apparatus includes: a texture determination module, used to determine an initial texture map of a first object image; each first object pixel in the first object image has texture coordinates, and the initial texture map stores the pixel coordinates of the first object pixel at the texture coordinates of the first object pixel; the initial texture map contains blank texture coordinates; a texture completion module, used to perform texture completion on the initial texture map to obtain a target texture map; the target texture map is obtained by storing the corresponding pixel coordinates at the blank texture coordinates in the initial texture map; a pose transformation module, used to perform pose feature transformation on a first feature map based on the texture coordinates of each second object pixel in a second object image and the target texture map to obtain a third feature map; the first feature map is obtained by feature extraction from the first object image; and an image generation module, used to perform clothing transformation based on the second feature map and the third feature map to generate a transformed object image corresponding to the second object image; the second feature map is obtained by feature extraction from the second object image.

[0008] In some embodiments, the texture determination module is configured to: determine the texture coordinates of each first object pixel in the first object image; for each first object pixel, store the pixel coordinates of the first object pixel at the texture coordinates of the first object pixel in the blank texture map, and generate an initial texture map corresponding to the first object image.

[0009] In some embodiments, the target texture map is obtained based on a trained texture completion network. The image processing device further includes a network training module. The steps of the network training module to obtain the trained texture completion network include: inputting an initial texture map of a third object image into the texture completion network to be trained for texture completion to obtain a current texture map; performing pose transformation on the third object image using the current texture map and the object pose information of a fourth object image to obtain a current pose transformation image; the third object image and the fourth object image include the same object with different poses; generating a first loss value based on the difference between the current pose transformation image and the fourth object image; and adjusting the network parameters of the texture completion network to be trained based on the first loss value to obtain the trained texture completion network.

[0010] In some embodiments, in adjusting the network parameters of the texture completion network to be trained based on a first loss value to obtain a trained texture completion network, the network training module is further configured to: restore the third object image using the object pose information of the current texture image and the third object image to obtain a current restored image; generate a second loss value based on the difference between the current restored image and the third object image; and adjust the network parameters of the texture completion network to be trained based on the first loss value and the second loss value to obtain a trained texture completion network.

[0011] In some embodiments, the image generation module is further configured to: fuse the features representing the target clothing in the third feature map into the features representing the target clothing in the second feature map to obtain a fused feature map; and generate a transformed object image corresponding to the second object image based on the object pose information corresponding to the second object image and the fused feature map.

[0012] In some embodiments, the pose transformation module is further configured to: for each first pixel coordinate in the first feature map, determine the texture coordinate of the second object pixel at the first pixel coordinate in the second object image to obtain the target texture coordinate corresponding to the first pixel coordinate; determine the pixel coordinate corresponding to the target texture coordinate in the target texture map to obtain the second pixel coordinate; update the feature value at the first pixel coordinate in the first feature map to the feature value at the second pixel coordinate in the first feature map to obtain the third feature map.

[0013] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the above-described image processing method.

[0014] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the above-described image processing method.

[0015] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described image processing method.

[0016] The aforementioned image processing method, apparatus, computer equipment, storage medium, and computer program product determine an initial texture map of the first object image. Since there are blank texture coordinates in the initial texture map, a target texture map is obtained by texture completion of the initial texture map. In the process of obtaining a third feature map by performing pose feature transformation on the first feature map based on the texture coordinates of each second object pixel in the second object image and the target texture map, the problem of missing pixel coordinates caused by the inconsistency of object poses in the first and second object images is reduced. This makes the effect of the transformed object image corresponding to the second object image generated by clothing transformation based on the second and third feature maps more realistic, thereby improving the effect of clothing transformation. Attached Figure Description

[0017] Figure 1 This is an application environment diagram of an image processing method in one embodiment;

[0018] Figure 2 This is a flowchart illustrating an image processing method in one embodiment;

[0019] Figure 3A This is a flowchart illustrating the texture feature extraction steps in one embodiment;

[0020] Figure 3B This is a flowchart illustrating the target object image generation steps in one embodiment;

[0021] Figure 4A This is a schematic diagram of the first object image in one embodiment;

[0022] Figure 4B This is a schematic diagram of the second object image in one embodiment;

[0023] Figure 5A This is a schematic diagram of the target texture map in one embodiment;

[0024] Figure 5B This is a schematic diagram of the second texture map in one embodiment;

[0025] Figure 5C This is a schematic diagram of the transformed object image in one embodiment;

[0026] Figure 6 This is a structural block diagram of an image processing device in one embodiment;

[0027] Figure 7 This is an internal structural diagram of a computer device in one embodiment;

[0028] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0030] The image processing method provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown includes a terminal 102 and a server 104, wherein the terminal 102 communicates with the server 104 via a network. A data storage system can store the data that the server 104 needs to process. The data storage system can be integrated onto the server 104, or it can be located in the cloud or on another network server.

[0031] Specifically, the terminal 102 or the server 104 may store a first object image and a second object image. In response to a virtual try-on operation, the terminal 102 sends a virtual try-on request to the server 104, the request carrying the first object image and the second object image. In response to the virtual try-on request, the server 104 determines an initial texture map of the first object image; each first object pixel in the first object image has texture coordinates, and the initial texture map stores the pixel coordinates of the first object pixels at their respective texture coordinate locations; the initial texture map contains blank texture coordinates. The server 104 performs texture completion on the initial texture map to obtain a target texture map; the target texture map is obtained by storing the corresponding pixel coordinates at the blank texture coordinate locations in the initial texture map. Then, based on the texture coordinates of each second object pixel in the second object image and the target texture map, the server 104 performs pose feature transformation on the first feature map to obtain a third feature map; the first feature map is obtained by feature extraction from the first object image. The server 104 performs clothing transformation based on the second and third feature maps to generate a transformed object image corresponding to the second object image; the second feature map is obtained by feature extraction from the second object image. Server 104 can send the transformed object image to terminal 102. Terminal 102 receives the transformed object image sent by server 104, displays and stores the transformed object image.

[0032] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0033] Those skilled in the art will understand that Figure 1The application environments shown are only some scenarios related to the solution of this application and do not constitute a limitation on the application environment of the solution of this application.

[0034] In some embodiments, such as Figure 2 As shown, an image processing method is provided, which can be executed by a terminal or a server, or by both a terminal and a server, and can be applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0035] Step 202: Determine the initial texture map of the first object image; each first object pixel in the first object image has texture coordinates; in the initial texture map, the texture coordinates of the first object pixel store the pixel coordinates of the first object pixel; there are blank texture coordinates in the initial texture map.

[0036] The object can be a person, and the object image is an image that includes the object; the object image is also called a model image. The first object image includes the first object, which can be obtained by photographing the first object or acquired from other devices. The first object in the first object image has target clothing, including tops, pants, hats, scarves, gloves, etc. The target clothing is the clothing to be transformed, and the first object image provides the image of the target clothing during the clothing transformation process; therefore, the first object image is also called the source model image.

[0037] The first object pixel is the pixel that makes up the first object region in the first object image. Each first object pixel in the first object image corresponds to a texture coordinate, which is used to represent its position in UV space, also called UV coordinate, and can be represented by (u, v). Pixel coordinates represent the position of the pixel in the image, and can be represented by (x, y). A texture map can accurately map each point on the image to the surface of a 3D model, and can also be called a UV map. In the initial texture map of the first object image, the texture coordinates of the first object pixels store the pixel coordinates of the first object pixels. The initial texture map contains blank texture coordinates, and the values ​​at blank texture coordinates are empty, for example, such as... Figure 4A As shown, Figure 4A Image (a) in the image is the first object. Figure 4A (b) in the image represents the first pose information (IUV). A , Figure 5A In the image (b), the initial texture map UV of the first object image is shown. A The first object image is obtained by taking a picture of the front of the first object. Therefore, the initial texture map of the first object image only includes the texture coordinates of the first object pixels on the front of the first object.

[0038] Specifically, the server can determine the texture coordinates of each pixel in the first object image, and then store the pixel coordinates of each first object pixel in the texture coordinates of the first object pixel in the blank texture map, generating the initial texture map corresponding to the first object image. Here, all texture coordinates in the blank texture map are blank texture coordinates, and the process of obtaining the initial texture map can also be called texture mapping or UV mapping.

[0039] In some embodiments, the server can perform pose estimation on the first object image to obtain first pose information, and use the first pose information to determine the texture coordinates of each pixel of the first object in the first object image. The first pose information represents the pose of the first object in the first object image. For example, if the first object image is A, the server can perform dense pose estimation on the first object image to obtain first pose information (IUV). A IUV A This includes the IUV coordinates (i, u, v) corresponding to each first object pixel (x, y), where i represents the human body part number to which the object pixel belongs, including the head, neck, arms, legs, etc., and u and v represent the texture coordinate values. Assuming the IUV coordinates corresponding to the first object pixel (x1, y1) are (i1, u1, v1), then the texture coordinates of the first object pixel (x1, y1) are (u1, v1). The generated initial texture map UV of the first object image... A In the texture coordinates (u1, v1), the pixel coordinates (x1, y1) are stored.

[0040] Step 204: Perform texture completion on the initial texture map to obtain the target texture map; the target texture map is obtained by storing the corresponding pixel coordinates at the blank texture coordinates in the initial texture map.

[0041] Texture completion involves determining the pixel coordinates corresponding to blank texture coordinates and storing those coordinates at the blank texture coordinate locations. The target texture map is obtained by completing the initial texture map of the first object image; blank texture coordinates are not present in the target texture map.

[0042] Specifically, the server can input the initial texture map of the first object image into a trained texture completion network for texture completion to obtain the target texture map, for example, Figure 5A In the image, (d) represents the target texture map UV of the first object image. A IThe trained texture completion network is capable of completing the pixel coordinates corresponding to blank texture coordinates in the initial texture map. These blank texture coordinates are derived from the pixel coordinates of the first object's pixel points stored in the initial texture map. For example, suppose the initial texture map UV of the first object image... A In the image, (u2, v2) represents the blank texture coordinates, and the UV coordinates of the target texture map obtained after texture completion. A I The texture coordinates (u2, v2) store the pixel coordinates (x1, y1).

[0043] In some embodiments, the server can first use human body symmetry to complete the initial texture map of the first object image to obtain a candidate texture map. Then, the candidate texture map is input into a trained texture completion network for texture completion to obtain the target texture map. For each blank texture coordinate in the initial texture map of the first object image, the server can determine the symmetrical texture coordinate from the initial texture map. If the symmetrical texture coordinate is not a blank texture coordinate, the pixel coordinate stored at the symmetrical texture coordinate is stored at the blank texture coordinate to obtain the candidate texture map. Here, the symmetrical texture coordinate refers to the texture coordinate that has a symmetrical relationship with the blank texture coordinate; for example, the symmetrical texture coordinate can be determined based on human body symmetry.

[0044] In some embodiments, the server may also determine an object mask image of the first object image to obtain the first object mask image, for example, such as... Figure 4A The first object mask image M shown in (d) is A And using the texture coordinates of each pixel of the first object, the mask texture map of the first object mask image is determined, for example, such as Figure 5A The mask texture map UV shown in (c) A M Then, the initial texture map of the first object image and the mask texture map of the first object mask image are input into the trained texture completion network for texture completion to obtain the target texture map. During the texture completion process, the mask texture map can be used to reduce the weight of texture pixels that store pixel coordinates in the initial texture map, thereby reducing changes to the pixel coordinates stored in the initial texture map and improving the accuracy of the target texture map.

[0045] In some embodiments, the server may also determine the texture coordinates of each second object pixel in the second object image. The specific process is the same as the steps described above for determining the texture coordinates of each first object pixel in the first object image, and will not be repeated here.

[0046] Step 206: Based on the texture coordinates of each second object pixel in the second object image and the target texture map, perform pose feature transformation on the first feature map to obtain the third feature map; the first feature map is obtained by feature extraction from the first object image.

[0047] The second object image includes a second object, which can be the same object as the first object or different objects. The pose of the second object in the second object image differs from the pose of the first object in the first object image. The second object image is an image of the target clothing obtained from the first object image during the clothing transformation process. For example, the first object image could be a model image obtained from an e-commerce platform, and the second object image could be a user selfie. The pixels of the second object are the pixels that make up the region of the second object in the second object image. The first feature map is obtained by extracting features from the first object image, and the third feature map is obtained by transforming the pose features of the first feature map. The size of the first feature map can be the same as the size of the first object image.

[0048] Specifically, the server extracts features from the first object image to obtain a first feature map, and then extracts features from the second object image to obtain a second feature map. For each first pixel coordinate in the first feature map, the server determines the texture coordinates of the second object pixel at the first pixel coordinate in the second object image based on the texture coordinates of each second object pixel in the second object image, thus obtaining the target texture coordinates corresponding to the first pixel coordinates. The server determines the pixel coordinates corresponding to the target texture coordinates from the target texture map to obtain the second pixel coordinates, and then updates the feature value at the first pixel coordinate in the first feature map to the feature value at the second pixel coordinate in the first feature map, thus obtaining a third feature map. Here, the first pixel coordinate can be any pixel coordinate in the first feature map. The target texture coordinates are determined based on the texture coordinates of the second object pixels in the second object image, and are the texture coordinates corresponding to the first pixel coordinates. The second pixel coordinates are the pixel coordinates corresponding to the target texture coordinates determined from the target texture map. The process of obtaining the third feature map can also be called texture inverse mapping or UV inverse mapping.

[0049] For example, suppose the coordinates of the first pixel are (x2, y2), and the first feature map F A 1 The eigenvalue at (x2, y2) is 0, representing the second pose information IUV. B The second object pixel (x2, y2) has IUV coordinates (i1, u2, v2), meaning the target texture coordinates of the first pixel (x2, y2) are (u2, v2). A IIn the texture coordinates (u2, v2), the stored pixel coordinates are (x3, y3), which means the second pixel coordinates are (x3, y3). Meanwhile, in the first feature map F... A 1 The feature value at (x3, y3) is 10, so the feature value at the first pixel coordinate (x2, y2) in the first feature map can be updated to the feature value of (x3, y3) in the first feature map, which is 10. This process continues until each first pixel coordinate in the first feature map is traversed to obtain the third feature map F. A 2 .

[0050] In some embodiments, the server can perform image stitching on the first object image and the first object mask image, and extract features from the stitched object image to obtain a first feature map. For example, assuming the scale of the first object image is (3, 1024, 768), that is, the first object image is a 3-channel 1024×768 image, and the scale of the first object mask image is (1, 1024, 768), performing a concat operation on the first object image and the first object mask image, the scale of the stitched object image is (4, 1024, 768).

[0051] Step 208: Perform clothing transformation based on the second feature map and the third feature map to generate the transformed object image corresponding to the second object image; the second feature map is obtained by extracting features from the second object image.

[0052] The second feature map is obtained by extracting features from the second object image. Clothing transformation refers to transforming the target clothing in the first object image into the second object image. The transformed object image includes the second object, and the pose of the second object in the transformed object image is consistent with the pose of the second object in the first object image. Furthermore, the target clothing of the target part of the second object in the target object image is consistent with the target clothing of the target part of the first object in the first object image. For example, if the target clothing is a top, such as... Figure 4A (a) and Figure 4B As shown in (a), a first object image and a second object image are displayed respectively. The first object in the first object image is wearing a vest, and the second object in the second object image is wearing a long-sleeved T-shirt. Figure 5C As shown, the transformed object image corresponding to the second object image is displayed, where the second object's top is transformed into a vest.

[0053] Specifically, the server fuses the features representing the target clothing in the third feature map into the features representing the target clothing in the second feature map to obtain a fused feature map. Then, based on the object pose information corresponding to the second object image and the fused feature map, the server generates the transformed object image corresponding to the second object image.

[0054] In some embodiments, the target texture map of the first object image is a first texture map. The server can also perform inverse texture mapping on the second feature map based on the texture coordinates of each second object pixel in the second object image and the second texture map of the second object image to obtain a fourth feature map. Then, based on the fourth feature map and the third feature map, clothing transformation is performed to generate the transformed object image corresponding to the second object image. The process of performing inverse texture mapping on the second feature map to obtain the fourth feature map can refer to the steps described above for obtaining the third feature map, and will not be repeated here. For example... Figure 5B (b) in the image represents the initial texture map UV of the second object image. B , Figure 5B In the image, (d) represents the target texture map UV of the second object image. A .

[0055] In some embodiments, the image processing method can be divided into two processes: determining texture features and generating an image. For example... Figure 3A The diagram illustrates the process of determining texture features. Both the first and second object images are object images. A dense pose estimation module obtains the pose information of the object image (i.e., the model image) and determines the texture coordinates of each object pixel. Then, a texture UV completion module maps the model image to its texture based on the texture coordinates of each object pixel, resulting in an initial texture map. This initial texture map is then completed to obtain the target texture map. A human semantic segmentation module obtains the human semantic segmentation results and the object mask image. A texture feature encoder extracts features from the model image and the object mask image. Based on the texture coordinates of each object pixel in the object image and the target texture map of the model image, the extracted feature maps are inversely mapped to their textures. Finally, a third and fourth feature map are obtained. The third feature map represents the source model texture, and the fourth feature map represents the target model texture. Figure 3B The diagram illustrates the process of image generation. By using a texture exchange model to exchange the textures of the source model and the target model with the textures of the target clothing, a fused feature map is obtained. Then, the image generation module uses the fused feature map and the second pose information to generate an image, resulting in the transformed object image.

[0056] In the above image processing method, an initial texture map of the first object image is determined. Since there are blank texture coordinates in the initial texture map, the target texture map is obtained by performing texture completion on the initial texture map. In the process of obtaining the third feature map by performing pose feature transformation on the first feature map based on the texture coordinates of each second object pixel in the second object image and the target texture map, the problem of missing pixel coordinates caused by the inconsistency of object poses in the first object image and the second object image is reduced. This makes the effect of the transformed object image corresponding to the second object image generated by clothing transformation based on the second feature map and the third feature map more realistic, thereby improving the effect of clothing transformation.

[0057] In some embodiments, step 202 includes: determining the texture coordinates of each first object pixel in the first object image; for each first object pixel, storing the pixel coordinates of the first object pixel at the texture coordinates of the first object pixel in the blank texture map, and generating an initial texture map corresponding to the first object image.

[0058] Specifically, the server can perform pose estimation on the first object image to obtain first pose information, determine the texture coordinates of each first object pixel from the first pose information, and then store the pixel coordinates of each first object pixel in the blank texture map at the texture coordinates of that first object pixel to generate the initial texture map corresponding to the first object image. The process of the server generating the initial texture map corresponding to the second object image can also refer to the above steps.

[0059] In this embodiment, by determining the texture coordinates of each first object pixel in the first object image and storing the pixel coordinates of each first object pixel in the texture coordinates of the first object pixel in the blank texture map, the initial texture map corresponding to the first object image is obtained quickly, which improves the efficiency of clothing transformation.

[0060] In some embodiments, the target texture map is obtained based on a trained texture completion network. The steps of obtaining the trained texture completion network include: inputting the initial texture map of the third object image into the texture completion network to be trained for texture completion to obtain the current texture map; performing pose transformation on the third object image using the object pose information of the current texture map and the fourth object image to obtain the current pose transformation image; the third object image and the fourth object image include the same object with different poses; generating a first loss value based on the difference between the current pose transformation image and the fourth object image; and adjusting the network parameters of the texture completion network to be trained based on the first loss value to obtain the trained texture completion network.

[0061] The trained texture completion network is used to complete the blank texture coordinates in the initial texture map. The texture completion network can be a U-Net network. The third and fourth object images include the same object in different poses, which can be obtained by taking pictures of the same object from different shooting angles. The third and fourth object images can be used as a training data pair. The third object image can be called the source, and the fourth object image can be called the target.

[0062] The initial texture map of the third object image contains blank texture coordinates. The current texture map is obtained by texture completion of the initial texture map of the third object image. The object pose information of the fourth object image is obtained by pose estimation of the fourth object image, including the texture coordinates of each object pixel in the fourth object image. The current pose transformation image is obtained by pose transformation of the third object image. The first loss value is determined based on the difference between the current pose transformation image and the fourth object image, and is used to adjust the network parameters of the texture completion network to be trained.

[0063] Specifically, the server can acquire a third object image and a fourth object image, and determine the initial texture map of the third object image and the object pose information of the fourth object image. The server inputs the initial texture map of the third object image into the texture completion network to be trained for texture completion, obtaining the current texture map. Using the current texture map and the object pose information of the fourth object image, the server performs pose transformation on the third object image, obtaining the current pose-transformed image. The server calculates the difference between the current pose-transformed image and the fourth object image to obtain a first loss value. Based on the first loss value, the server adjusts the network parameters of the texture completion network to be trained until the network converges, obtaining the trained texture completion network. The process of determining the initial texture map of the third object image can refer to the steps described above for determining the processed texture map of the first object image, and the process of performing pose transformation on the third object image can refer to the steps described above for obtaining the third feature map, and will not be repeated here.

[0064] In this embodiment, since the third object image and the fourth object image include the same object with different poses, the current pose transformation image is obtained by transforming the pose of the third object image using the object pose information of the current texture image and the fourth object image. In order to generate a first loss value by calculating the difference between the current pose transformation image and the fourth object image, and to adjust the network parameters of the texture completion network to be trained using the first loss value, the trained texture completion network has the function of completing the pixel coordinates corresponding to the blank texture coordinates in the initial texture image, thereby improving the accuracy of texture completion.

[0065] In some embodiments, adjusting the network parameters of the texture completion network to be trained based on a first loss value to obtain a trained texture completion network includes: restoring the third object image using the object pose information of the current texture image and the third object image to obtain a current restored image; generating a second loss value based on the difference between the current restored image and the third object image; and adjusting the network parameters of the texture completion network to be trained based on the first loss value and the second loss value to obtain a trained texture completion network.

[0066] The object pose information of the third object image is obtained by pose estimation of the third object image, including the texture coordinates of each object pixel in the third object image. The current reconstructed image is an image obtained using the object pose information of the current texture map and the third object image. The second loss value is obtained based on the difference between the current reconstructed image and the third object image.

[0067] Specifically, the server can also determine the object pose information of the third object image, use the current texture map and the object pose information of the third object image to reconstruct the third object image, obtain the current reconstructed image, then calculate the difference between the current reconstructed image and the third object image to obtain the second loss value, and use the first and second loss values ​​to adjust the network parameters of the texture completion network to be trained. Then the server can update the third object image and the fourth object image, return to execute the above steps, until the network converges, and obtain the trained texture completion network.

[0068] In this embodiment, the first loss value is determined based on the difference between the current pose transformation image and the fourth object image, and the second loss value is determined based on the difference between the current restored image and the third object image. By using the first loss value and the second loss value together to adjust the network parameters of the texture completion network to be trained, the accuracy of texture completion is further improved, thereby using the accurate target texture map to perform pose feature transformation on the first feature map, and the resulting third feature map is also more accurate.

[0069] In some embodiments, step 208 further includes: fusing the features representing the target clothing in the third feature map into the features representing the target clothing in the second feature map to obtain a fused feature map; and generating a transformed object image corresponding to the second object image based on the object pose information corresponding to the second object image and the fused feature map.

[0070] Specifically, the server replaces the features representing the target clothing in the third feature map with the features representing the target clothing in the second feature map, obtaining a fused feature map. Then, the server encodes the object pose information corresponding to the second object image to obtain target pose features; then, it encodes the fused feature map to obtain target texture features. The server then inputs the target texture features and target pose features into a trained image generation network to generate a transformed object image corresponding to the second object image. Here, the target texture features and target pose features are used to generate the target object image. During image generation, the target texture features control the texture representation of the object in the target object image, and the target pose features control the pose representation of the object in the target object image. The trained image generation network takes the target texture features and target pose features as input and has the function of generating realistic object images.

[0071] For example, the trained image generation network could be a conditional StyleGAN2 generation network, utilizing a pose encoder to process the second pose information (IUV). A Encoding is performed to obtain the target pose features, also known as the latent vector ω; a texture encoder is then used to process the fused texture map F. in Encoding is performed to obtain the target texture features. Then, the target pose features and target texture features are input into the conditional StyleGAN2 generator network to generate a realistic transformed object image I′. B .

[0072] In some embodiments, the server can determine a mask image of the target clothing in the second object image, obtaining a target clothing mask image. The size of the target clothing mask image is consistent with the size of the third feature map and the size of the second feature map. Features representing the target clothing are determined from the third feature map using the target clothing mask image. Then, the features representing the target clothing in the third feature map replace the features representing the target clothing in the second feature map, resulting in a fused feature map. For example, the server can perform human semantic segmentation on the second object image to determine the target clothing mask image. For example, assuming the target clothing is a top, such as... Figure 4B As shown in (c), the result P of human semantic segmentation on the second object image is presented. B It can be based on P B The upper garment area in the image is used to determine the target clothing mask image M. up The process of obtaining the fused feature map can be represented by the following formula:

[0073]

[0074] Among them, F in M represents the fused feature map. upThe target clothing mask image is shown, and ⊙ represents element-wise multiplication at corresponding positions in a matrix operation. This is the third feature map. This is the second feature map.

[0075] In this embodiment, since the third feature map is obtained by transforming the first feature map into pose features, the pose represented by the third feature map is consistent with the pose represented by the second feature map. The features representing the target clothing in the third feature map are fused into the features representing the target clothing in the second feature map, resulting in a more accurate fused feature map. Therefore, based on the object pose information corresponding to the second object image and the fused feature map, the transformed object image corresponding to the second object image is generated, which improves the generation effect of the transformed object image, and the object pose remains unchanged.

[0076] In some embodiments, step 206 further includes: for each first pixel coordinate in the first feature map, determining the texture coordinates of the second object pixel at the first pixel coordinate in the second object image to obtain the target texture coordinates corresponding to the first pixel coordinates; determining the pixel coordinates corresponding to the target texture coordinates in the target texture map to obtain the second pixel coordinates; updating the feature value at the first pixel coordinate in the first feature map to the feature value at the second pixel coordinate in the first feature map to obtain the third feature map.

[0077] Specifically, the server determines the texture coordinates of each second object pixel based on the second pose information of the second object image. Then, for each first pixel coordinate in the first feature map, the server can use the texture coordinates of each second object pixel to determine the texture coordinates of the second object pixel at the first pixel coordinate in the second object image, thus obtaining the target texture coordinates corresponding to the first pixel coordinates. The server can then determine the pixel coordinates corresponding to the target texture coordinates from the target texture map, obtain the second pixel coordinates, and update the feature value at the first pixel coordinate in the first feature map with the feature value at the second pixel coordinate in the first feature map, thus obtaining the third feature map.

[0078] In this embodiment, by utilizing the texture coordinates of each second object pixel in the second object image, the target texture coordinates corresponding to each first pixel coordinate in the first feature map are determined. Then, using the target texture map and target texture coordinates of the first object image, the second pixel coordinates are determined. Finally, the feature value at the first pixel coordinate in the first feature map is updated to the feature value at the second pixel coordinate in the first feature map to obtain the third feature map. This achieves pose feature transformation of the first feature map, making the features representing the object pose in the third feature map consistent with the features representing the object pose in the second feature map of the second object image. This reduces the problem of unrealistic clothing transformation caused by inconsistent poses, thereby improving the effect of clothing transformation.

[0079] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0080] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0081] In some embodiments, such as Figure 6 As shown, an image processing apparatus is provided, including: a texture determination module 602, a texture completion module 604, a pose transformation module 606, and an image generation module 608, wherein:

[0082] The texture determination module 602 is used to determine the initial texture map of the first object image; each first object pixel in the first object image has texture coordinates; in the initial texture map, the texture coordinates of the first object pixel store the pixel coordinates of the first object pixel; there are blank texture coordinates in the initial texture map.

[0083] The texture completion module 604 is used to complete the initial texture map to obtain the target texture map; the target texture map is obtained by storing the corresponding pixel coordinates at the blank texture coordinates in the initial texture map.

[0084] The pose transformation module 606 is used to perform pose feature transformation on the first feature map to obtain a third feature map based on the texture coordinates of each second object pixel in the second object image and the target texture map; the first feature map is obtained by feature extraction from the first object image.

[0085] The image generation module 608 is used to perform clothing transformation based on the second feature map and the third feature map to generate the transformed object image corresponding to the second object image; the second feature map is obtained by extracting features from the second object image.

[0086] In some embodiments, the texture determination module 602 is used to: determine the texture coordinates of each first object pixel in the first object image; and for each first object pixel, store the pixel coordinates of the first object pixel in the texture coordinates of the first object pixel in the blank texture map to generate an initial texture map corresponding to the first object image.

[0087] In some embodiments, the target texture map is obtained based on a trained texture completion network. The image processing apparatus further includes a network training module. The steps of the network training module to obtain the trained texture completion network include: inputting an initial texture map of a third object image into the texture completion network to be trained for texture completion to obtain a current texture map; performing pose transformation on the third object image using the object pose information of the current texture map and the fourth object image to obtain a current pose transformation image; the third object image and the fourth object image include the same object with different poses; generating a first loss value based on the difference between the current pose transformation image and the fourth object image; and adjusting the network parameters of the texture completion network to be trained based on the first loss value to obtain the trained texture completion network.

[0088] In some embodiments, in adjusting the network parameters of the texture completion network to be trained based on the first loss value to obtain a trained texture completion network, the network training module is further configured to: restore the third object image using the object pose information of the current texture image and the third object image to obtain a current restored image; generate a second loss value based on the difference between the current restored image and the third object image; and adjust the network parameters of the texture completion network to be trained based on the first loss value and the second loss value to obtain a trained texture completion network.

[0089] In some embodiments, the image generation module 608 is further configured to: fuse the features representing the target clothing in the third feature map into the features representing the target clothing in the second feature map to obtain a fused feature map; and generate a transformed object image corresponding to the second object image based on the object pose information corresponding to the second object image and the fused feature map.

[0090] In some embodiments, the pose transformation module 606 is further configured to: for each first pixel coordinate in the first feature map, determine the texture coordinates of the second object pixel at the first pixel coordinate in the second object image to obtain the target texture coordinates corresponding to the pixel coordinates; determine the pixel coordinates corresponding to the target texture coordinates in the target texture map to obtain the second pixel coordinates; update the feature value at the first pixel coordinate in the first feature map to the feature value at the second pixel coordinate in the first feature map to obtain the third feature map.

[0091] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0092] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores relevant data related to the image processing method. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an image processing method.

[0093] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0094] Those skilled in the art will understand that Figure 7 and Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0095] In some embodiments, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the image processing method described above.

[0096] In some embodiments, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the image processing method described above.

[0097] In some embodiments, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the image processing method described above.

[0098] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0099] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0100] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: An initial texture map of the first object image is determined; each first object pixel in the first object image has texture coordinates; in the initial texture map, the texture coordinates of the first object pixel store the pixel coordinates of the first object pixel; the initial texture map contains blank texture coordinates; The initial texture map is filled with texture to obtain the target texture map; the target texture map is obtained by storing the corresponding pixel coordinates at the blank texture coordinates in the initial texture map. For each first pixel coordinate in the first feature map, the texture coordinates of the second object pixel at the first pixel coordinate are determined from the second object image to obtain the target texture coordinates corresponding to the first pixel coordinates; the first feature map is obtained by feature extraction from the first object image. The second pixel coordinates are obtained by determining the pixel coordinates corresponding to the target texture coordinates from the target texture map; The feature value at the first pixel coordinate in the first feature map is updated to the feature value at the second pixel coordinate in the first feature map to obtain the third feature map; The features representing the target clothing in the third feature map are fused into the features representing the target clothing in the second feature map to obtain a fused feature map; the second feature map is obtained by feature extraction from the second object image. Based on the object pose information corresponding to the second object image and the fused feature map, a transformed object image corresponding to the second object image is generated.

2. The method according to claim 1, characterized in that, The determination of the initial texture map of the first object image includes: Determine the texture coordinates of each pixel point of the first object in the first object image; For each pixel of the first object, the pixel coordinates of the first object pixel are stored in the texture coordinates of the first object pixel in the blank texture map to generate the initial texture map corresponding to the first object image.

3. The method according to claim 1, characterized in that, The target texture map is obtained based on a trained texture completion network. The steps for obtaining the trained texture completion network include: The initial texture map of the third object image is input into the texture completion network to be trained for texture completion, and the current texture map is obtained. The pose transformation of the third object image is performed using the object pose information of the current texture map and the fourth object image to obtain the current pose transformation image; the third object image and the fourth object image include the same object with different poses; A first loss value is generated based on the difference between the current pose transformation image and the fourth object image; The network parameters of the texture completion network to be trained are adjusted based on the first loss value to obtain the trained texture completion network.

4. The method according to claim 3, characterized in that, The step of adjusting the network parameters of the texture completion network to be trained based on the first loss value to obtain the trained texture completion network includes: The third object image is restored using the current texture map and the object pose information of the third object image to obtain the current restored image; A second loss value is generated based on the difference between the current restored image and the third object image; Based on the first loss value and the second loss value, the network parameters of the texture completion network to be trained are adjusted to obtain the trained texture completion network.

5. An image processing apparatus, characterized in that, The device includes: A texture determination module is used to determine an initial texture map of a first object image; each first object pixel in the first object image has texture coordinates, and the initial texture map stores the pixel coordinates of the first object pixel at the texture coordinates of the first object pixel; the initial texture map contains blank texture coordinates; The texture completion module is used to complete the initial texture map to obtain a target texture map; the target texture map is obtained by storing the corresponding pixel coordinates at the blank texture coordinates in the initial texture map. The pose transformation module is used to perform pose feature transformation on the first feature map to obtain a third feature map based on the texture coordinates of each second object pixel in the second object image and the target texture map; the first feature map is obtained by feature extraction from the first object image. The image generation module is used to perform clothing transformation based on the second feature map and the third feature map to generate a transformed object image corresponding to the second object image; the second feature map is obtained by extracting features from the second object image. The pose transformation module is further configured to, for each first pixel coordinate in the first feature map, determine the texture coordinates of the second object pixel at the first pixel coordinate in the second object image, thereby obtaining the target texture coordinates corresponding to the first pixel coordinates; the first feature map is obtained by feature extraction from the first object image; the pixel coordinates corresponding to the target texture coordinates are determined from the target texture map to obtain the second pixel coordinates; the feature value at the first pixel coordinate in the first feature map is updated to the feature value at the second pixel coordinate in the first feature map to obtain the third feature map; The image generation module is further configured to fuse the features representing the target clothing in the third feature map into the features representing the target clothing in the second feature map to obtain a fused feature map; the second feature map is obtained by feature extraction of the second object image; and based on the object pose information corresponding to the second object image and the fused feature map, a transformed object image corresponding to the second object image is generated.

6. The apparatus according to claim 5, characterized in that, The texture determination module is further configured to determine the texture coordinates of each first object pixel in the first object image; for each first object pixel, the pixel coordinates of the first object pixel are stored in the texture coordinates of the first object pixel in the blank texture map to generate an initial texture map corresponding to the first object image.

7. The apparatus according to claim 5, characterized in that, The texture completion module is further configured to input the initial texture map of the third object image into the texture completion network to be trained for texture completion to obtain the current texture map; use the object pose information of the current texture map and the fourth object image to perform pose transformation on the third object image to obtain the current pose transformation image; the third object image and the fourth object image include the same object with different poses; and generate a first loss value based on the difference between the current pose transformation image and the fourth object image. The network parameters of the texture completion network to be trained are adjusted based on the first loss value to obtain the trained texture completion network.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images