Target object dressing method and device, electronic equipment, storage medium and computer program product

Abstract

This application provides a method, apparatus, electronic device, storage medium, and computer program product for changing the clothing of a target object. The method includes: acquiring an image to be changed into and a clothing image; the image to be changed into includes a target object; determining a semantic segmentation map of the image to be changed into and a pose feature map of the target object; performing masking processing on the image to be changed into based on the pose feature map and the semantic segmentation map to obtain a masked image to be changed into; extracting texture features from the clothing image to obtain the texture features of the target clothing in the clothing image; determining a changing image based on the pose feature map, the masked image to be changed into, and the texture features of the target clothing; the changing image includes the target object wearing the target clothing. This application generates a changing image by extracting the texture features of the target clothing and combining them with the texture features of the clothing features, so that the clothing in the changing image retains the real texture of the clothing, thereby improving the fidelity and realism of the changing image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and includes, but is not limited to, a method, apparatus, electronic device, storage medium, and computer program product for changing the appearance of a target object. Background Technology

[0002] With the development of the internet and online shopping platforms, online clothing shopping has seen significant growth. Compared with traditional shopping, online shopping has many advantages such as convenience and speed. However, since it is not possible to try on and inspect clothing in real time like in physical stores before purchasing, most people still choose to buy clothing in physical stores. In order to improve the online shopping experience for users, virtual fitting methods have emerged.

[0003] In related technologies, most common virtual try-on algorithms use a distortion and pasting method to blend the model and the clothing. However, virtual try-on images generated by distortion and pasting ignore the real clothing folds and natural lighting, resulting in low fidelity and realism. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, storage medium, and computer program product for changing the clothing of a target object. It can be applied to at least the field of artificial intelligence. This application generates a changing image by extracting the texture features of the target clothing and combining the texture features of the clothing features, so that the clothing in the changing image retains the real clothing texture, thereby improving the fidelity and realism of the changing image.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a method for changing the clothing of a target object, comprising: acquiring an image to be changed into and a clothing image; the image to be changed into includes a target object; determining a semantic segmentation map of the image to be changed into and a pose feature map of the target object; performing masking processing on the image to be changed into based on the pose feature map and the semantic segmentation map to obtain a masked image to be changed into; extracting texture features from the clothing image to obtain the texture features of the target clothing in the clothing image; determining a changing image based on the pose feature map, the masked image to be changed into, and the texture features of the target clothing; the changing image includes the target object wearing the target clothing.

[0007] This application provides a target object dressing device, comprising: an acquisition module for acquiring an image to be dressed and a clothing image; the image to be dressed includes a target object; a first determination module for determining a semantic segmentation map of the image to be dressed and a pose feature map of the target object; a masking module for masking the image to be dressed based on the pose feature map and the semantic segmentation map to obtain a masked image to be dressed; a feature extraction module for extracting texture features from the clothing image to obtain texture features of the target clothing in the clothing image; and a second determination module for determining a dressing image based on the pose feature map, the masked image to be dressed, and the texture features of the target clothing; the dressing image includes a target object wearing the target clothing.

[0008] In the above scheme, the first determining module is further configured to: preprocess the image to be changed into a standard image to be changed into; perform image segmentation on the standard image to be changed into a pre-trained image segmentation model to obtain an image segmentation result containing multiple human body regions and multiple background regions; and classify the pixels in the standard image to be changed into a semantic segmentation map of the image to be changed into based on the image segmentation result.

[0009] In the above scheme, the first determining module is further configured to: perform key point detection on the target object in the standard image to be dressed up using a pre-trained key point detection model, obtain multiple key point coordinates of the target object, and key point confidence corresponding to each key point coordinate; filter the key point coordinates based on the key point confidence to obtain target key point coordinates; and mark the target key points corresponding to the target key point coordinates on the standard image to be dressed up based on the target key point coordinates to obtain the pose feature map of the target object.

[0010] In the above scheme, the semantic segmentation map includes label information for each region in the image to be dressed up; the region includes the human body region and the background region; the masking module is further configured to: determine a target region from the semantic segmentation map based on the label information, and generate a binary image of the target region; adjust the contour of the binary image based on the key point coordinates in the pose feature map to obtain an adjusted binary image; smooth the adjusted binary image to obtain an optimized binary image; align the optimized binary image with the standard image to be dressed up to obtain an aligned image; and based on the aligned image, replace the pixel values ​​in the occluded region corresponding to the optimized binary image with the region corresponding to the occluded region in the standard image to be dressed up to obtain the masked image to be dressed up.

[0011] In the above scheme, the feature extraction module is further configured to: extract global texture features from the clothing image using a pre-trained image encoder to obtain global texture features of the target clothing in the clothing image; extract local texture features from the clothing image using a pre-trained texture feature extraction model to obtain local texture features of the target clothing; and determine the global texture features and the local texture features as the texture features of the target clothing.

[0012] In the above scheme, the second determining module is further configured to: extract features from the posture feature map using a pre-trained posture encoder to obtain a posture feature vector; obtain random Gaussian noise; stitch the random Gaussian noise and the masked image to be changed in the channel dimension to generate a stitched image tensor; extract features from the stitched image tensor to obtain a stitched image vector; and determine the changing image based on the stitched image vector, the posture feature vector, and the texture features of the target clothing.

[0013] In the above scheme, the texture features of the target clothing include global texture features and local texture features; the second determining module is further configured to: add the stitched image vector and the pose feature vector to obtain an input vector; input the input vector, the global texture features and the local texture features into a pre-trained main network model to obtain a dressing image; wherein, the pre-trained main network model includes multiple data processing modules connected in sequence; the input vector is input into the first data processing layer of the first data processing module in the pre-trained main network model, the global texture features are input into the cross-attention layer of each data processing module in the pre-trained main network model, and the local texture information is input into the self-attention mechanism layer of each data processing module in the pre-trained main network model.

[0014] In the above scheme, the acquisition module is further configured to: acquire the original image to be changed into and the original clothing image; preprocess the original image to be changed into and the original clothing image to obtain the preprocessed original image to be changed into and the preprocessed original clothing image; remove the background from the preprocessed original image to be changed into and the preprocessed original clothing image using a pre-trained background removal model to obtain the image to be changed into and the clothing image; the background area in the image to be changed into and the clothing image is transparent or solid color.

[0015] In the above scheme, the original clothing image includes the target clothing and the original target object wearing the target clothing; the device further includes a processing module, used to perform image segmentation on the original clothing image using a pre-trained image segmentation model to obtain a segmentation mask; based on the segmentation mask, to delete the background region and human body structure region in the original clothing image to obtain an original clothing image including the target clothing region; and to perform feature enhancement on the target clothing region to obtain the preprocessed original clothing image.

[0016] This application provides an electronic device, including: a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to implement the above-described target object replacement method.

[0017] This application provides a computer program product that stores computer-executable instructions or a computer program, which is used to cause a processor to execute the computer-executable instructions or the computer program to implement the above-mentioned target object replacement method.

[0018] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs, which are used to cause a processor to execute the computer-executable instructions or computer programs to implement the above-described target object replacement method.

[0019] The above scheme has the following beneficial effects:

[0020] The target object dressing method in this embodiment generates a semantic segmentation map and a pose feature map based on the image to be dressed, which can accurately identify and separate the body regions and pose information of the target object. Moreover, by masking the image to be dressed, the interference of the background on the dressing is effectively avoided, so that the dressing effect is focused on the target object itself. Furthermore, by extracting texture features from the clothing image, the texture features of the target clothing are obtained, and the dressing image is generated by combining the texture features, so that the clothing in the dressing image retains the real clothing texture, thereby improving the fidelity and realism of the dressing image, enhancing the visual realism, and improving the user experience. Attached Figure Description

[0021] Figure 1 This is an optional architecture diagram of the target object dressing system provided in the embodiments of this application;

[0022] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0023] Figure 3 This is an optional flowchart illustrating the target object replacement method provided in the embodiments of this application;

[0024] Figure 4 This is another optional flowchart illustrating the target object replacement method provided in the embodiments of this application;

[0025] Figure 5 This is a schematic diagram of a semantic segmentation map for determining an image to be dressed up, provided in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the pose feature map of the target object provided in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of a masked image of an image to be dressed up, obtained by masking an image based on a pose feature map and a semantic segmentation map according to an embodiment of this application.

[0028] Figure 8 This is a flowchart illustrating the image to be dressed after the mask has been determined, as provided in an embodiment of this application.

[0029] Figure 9 This is a schematic diagram of the process of determining the dressing image based on the pose feature map, the masked image to be dressed, and the texture features of the target garment, provided in the embodiments of this application;

[0030] Figure 10 This is a schematic diagram of the input of the first data processing module in the pre-trained main network model provided in the embodiments of this application;

[0031] Figure 11 This is a schematic diagram of the training process of the original costume-changing network provided in the embodiments of this application;

[0032] Figure 12 This is a rendering of the costume change image provided in the embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application.

[0035] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0036] To better understand the target object replacement method provided in the embodiments of this application, the target object replacement method in related technologies will be described below.

[0037] In related technologies, most target object dressing methods use a warp and paste approach to blend the model and clothing. Taking the classic virtual try-on network (VITON) as an example: VITON employs a two-step synthesis and refinement process. First, a rough image with the target clothing is initially generated, and then the generated rough image is refined to enhance detailed features. Specifically, VITON first creates a clothing-independent character representation, which includes features such as the character's pose, body parts, face, and hair. These features are used as input to the generator to constrain the synthesis process. Next, VITON proposes a multi-task encoder-decoder generator. Given the character representation and the target clothing image, VITON uses a multi-task encoder-decoder framework to generate a synthetic image of the character wearing the target clothing, along with a corresponding clothing region mask. The clothing region mask serves as guidance information, adapting to the deformation of the clothing through deformation. Finally, to render more detailed visual patterns in the coarse synthetic result, VITON uses a refinement network trained to learn how to fuse the deformed target clothing image with the coarse synthetic image, outputting the final try-on result.

[0038] However, the above-mentioned methods for changing the appearance of the target object have at least the following problems:

[0039] 1) Related technologies rely too much on the display distortion process, ignoring real clothing folds and natural lighting, which seriously reduces the fidelity and realism of the synthesized images; 2) Most existing virtual try-on algorithms are trained based on Generative Adversarial Networks (GANs), but GAN-based models suffer from overfitting of training data. Therefore, the try-on effect will be severely degraded and the generalization ability will be poor in scenarios that are not within the training set distribution.

[0040] To address the problems existing in related technologies, this application provides a method for changing the clothing of a target object. It generates a semantic segmentation map and a pose feature map based on the image to be changed, enabling accurate identification and separation of the target object's body regions and pose information. Furthermore, by masking the image to be changed, background interference is effectively avoided, allowing the changing effect to focus on the target object itself. Additionally, by extracting texture features from the clothing image, the texture features of the target clothing are obtained, and the changing image is generated in combination with these texture features. This ensures that the clothing in the changing image retains its realistic texture, thereby improving the fidelity and realism of the changing image, enhancing the visual realism, and improving the user experience.

[0041] Specifically, this application provides a method for changing the clothing of a target object. First, an image to be changed into clothing and an image of clothing are acquired; the image to be changed into clothing includes a target object. Then, a semantic segmentation map of the image to be changed into clothing and a pose feature map of the target object are determined. Next, based on the pose feature map and the semantic segmentation map, the image to be changed into clothing is masked to obtain a masked image to be changed into clothing. Then, texture features are extracted from the clothing image to obtain the texture features of the target clothing in the clothing image. Finally, based on the pose feature map, the masked image to be changed into clothing, and the texture features of the target clothing, a changing image is determined; the changing image includes a target object wearing the target clothing.

[0042] Here, we first describe an exemplary application of the target object dressing device in this application embodiment. This target object dressing device is an electronic device used to implement the target object dressing method. In one implementation, the target object dressing device (i.e., electronic device) provided in this application embodiment can be implemented as a terminal or as a server. In one implementation, the electronic device provided in this application embodiment can be implemented as any terminal with target object dressing function, such as a laptop, tablet, desktop computer, or intelligent robot. In another implementation, the target object dressing device provided in this application embodiment can also be implemented as a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in this application embodiment. Below, we will describe an exemplary application when the target object dressing device is implemented as a server.

[0043] See Figure 1 , Figure 1 This is an optional architecture diagram of the target object replacement system provided in this application embodiment. The target object replacement system 10 in this application embodiment includes at least a terminal 100, a network 200, and a server 300. The terminal 100 is deployed with a target object replacement application, and the server 300 can be a backend server for the target object replacement application. The server 300 can constitute the target object replacement device in this application embodiment, that is, the target object replacement method in this application embodiment is implemented through the server 300. The terminal 100 is connected to the server 300 through the network 200, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0044] See Figure 1Terminal 100 receives a user's clothing change operation on the image to be changed, and in response to the operation, generates a clothing change request. Then, terminal 100 sends the clothing change request to server 300 via network 200. Upon receiving the clothing change request, server 300, in response, acquires the original image to be changed and the original clothing image, and performs image optimization on both to obtain the image to be changed and the clothing image, wherein the image to be changed includes the target object. Next, server 300 determines the semantic segmentation map of the image to be changed and the pose feature map of the target object. Next, server 300 performs masking processing on the image to be dressed up based on the pose feature map and semantic segmentation map to obtain the masked image to be dressed up. Then, server 300 extracts texture features from the clothing image to obtain the texture features of the target clothing in the clothing image. Next, server 300 determines the dressing up image based on the pose feature map, the masked image to be dressed up, and the texture features of the target clothing. The dressing up image includes the target object wearing the target clothing. Finally, server 300 sends the dressing up image to terminal 100 through network 200, and displays the dressing up image on the display interface of terminal 100.

[0045] In some embodiments, the above-described target object dressing method can also be executed by a terminal. That is, after receiving a user's dressing operation on the image to be dressed, the terminal 100 can acquire the original image to be dressed and the original clothing image, and perform image optimization on the original image to be dressed and the original clothing image to obtain the image to be dressed and the clothing image, wherein the image to be dressed includes the target object; then, the terminal 100 determines the semantic segmentation map of the image to be dressed and the pose feature map of the target object; next, the terminal 100 performs masking processing on the image to be dressed based on the pose feature map and the semantic segmentation map to obtain the masked image to be dressed; next, the terminal 100 extracts texture features from the clothing image to obtain the texture features of the target clothing in the clothing image; next, the terminal 100 determines the dressing image based on the pose feature map, the masked image to be dressed, and the texture features of the target clothing; the dressing image includes the target object wearing the target clothing; finally, the terminal 100 displays the dressing image on the display interface.

[0046] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 2The illustrated electronic device may be a target object replacement device, which includes at least one processor 210, a memory 250, at least one network interface 220, and a user interface 230. The various components in the target object replacement device are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 240.

[0047] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0048] User interface 230 includes one or more output devices 231 that enable the presentation of media content, and one or more input devices 232.

[0049] Memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 250 may optionally include one or more storage devices physically located remote from processor 210. Memory 250 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 250 described in this application embodiment is intended to include any suitable type of memory. In some embodiments, memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as illustrated below.

[0050] Operating system 251 includes system programs for handling various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; network communication module 252 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 include: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.; input processing module 253 is used to detect and translate one or more user inputs or interactions from one or more input devices 232.

[0051] In some embodiments, the apparatus provided in this application may be implemented in software. Figure 2 A target object replacement device 254 stored in memory 250 is shown. This target object replacement device 254 can be a target object replacement device in an electronic device, and can be software in the form of programs and plug-ins, including the following software modules: an acquisition module 2541, a first determination module 2542, a mask module 2543, a feature extraction module 2544, and a second determination module 2545. These modules are logically connected and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0052] In some embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the target object replacement method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0053] The target object replacement methods provided in the embodiments of this application can be executed by an electronic device, which can be a server or a terminal. That is, the target object replacement methods in the embodiments of this application can be executed by a server, by a terminal, or by interaction between a server and a terminal.

[0054] Figure 3 This is an optional flowchart illustrating the target object replacement method provided in the embodiments of this application. The following will be combined with... Figure 3 The steps shown are explained as follows: Figure 3 As shown, taking the server as the execution subject of the target object replacement method as an example, the method includes the following steps S101 to S105:

[0055] Step S101: Obtain the image of the outfit to be changed and the image of the clothing; the image of the outfit to be changed includes the target object.

[0056] An image to be changed refers to an image containing the target object that needs to be changed, usually a complete image in a certain pose; the target object is the main subject in the image to be changed, usually referring to the person or object that needs to be dressed up; the clothing image refers to the image of the new clothing to be replaced, usually displayed in a tiled form, and the clothing image usually needs to have a certain angle and clarity.

[0057] In some embodiments, the image of the outfit to be changed and the image of the clothing can be obtained in any of the following ways:

[0058] Method 1: Select the images of the clothes to be changed and the clothing images from a pre-saved image database.

[0059] Method 2: Users take real-time photos of the target object and the clothing to be replaced using a camera, and use the captured images as images of the clothing to be changed and the clothing itself.

[0060] Method 3: Users can manually upload images of the outfit to be changed and the clothing itself.

[0061] Step S102: Determine the semantic segmentation map of the image to be dressed up and the pose feature map of the target object.

[0062] A semantic segmentation map is an image obtained by classifying the pixels of an image to be dressed up and labeling the category to which each pixel belongs. Semantic segmentation maps are used to separate the target object and its different parts from the background in an image to be dressed up. A pose feature map is a map that contains the limb position and pose information of the target object, and is usually composed of a set of key points and skeleton lines.

[0063] In some instances, the semantic segmentation map of the image to be dressed up can be determined by the following method: First, the image to be dressed up is preprocessed to obtain a standard image to be dressed up; then, a suitable pre-trained image segmentation model, such as UNet or DeepLab, is selected, and the standard image to be dressed up is input into the pre-trained image segmentation model to generate the category label of each pixel in the image to be dressed up, thus obtaining the semantic segmentation map.

[0064] In some embodiments, the pose feature map of the target object can be determined as follows: First, the image to be dressed up is preprocessed to obtain a standard image to be dressed up; then, a suitable pre-trained keypoint detection model is selected, such as OpenP. Using algorithms like Osose and HRNet, keypoint detection is performed on the target object in the standard image to be dressed up, obtaining the coordinates of multiple keypoints and the confidence score of each keypoint. Then, the confidence scores are used to filter each keypoint coordinate to obtain the target keypoints. For example, if there are five keypoint coordinates: coordinates 1, 2, 3, 4, and 5, with a confidence score of 0.9 for coordinate 1, 0.4 for coordinate 2, 0.85 for coordinate 3, 0.6 for coordinate 4, and 0.95 for coordinate 5, and a pre-set confidence score of 0.8, then the target keypoint coordinates are determined to be coordinates 1, 3, and 5. Finally, the corresponding positions of the target keypoint coordinates in the standard image to be dressed up are marked to generate a skeleton map, which is the pose feature map of the target object.

[0065] Step S103: Based on the pose feature map and semantic segmentation map, the image to be dressed up is masked to obtain the masked image to be dressed up.

[0066] Masking refers to the operation of covering or extracting a specified area in an image. Unrelated areas can be hidden through masking. The image to be dressed up after masking is the image obtained after the image to be dressed up has been masked.

[0067] In some embodiments, step S103 can be implemented by the following method: First, a binary mask is generated using a semantic segmentation map to mark the area to be dressed up, such as marking the pixels of the upper body area of ​​the target object as "1", while the background and other areas that do not need to be dressed up are marked as "0"; then, the mask edges are optimized using key point information in the pose feature map so that the outline of the binary mask matches the specific pose of the target object, especially applying smooth transitions at joints and body contours to make the binary mask fit the target pose better; finally, the mask is applied to the standard image to be dressed up to obtain the masked image to be dressed up.

[0068] It should be noted that in step S103, the semantic segmentation map and pose feature map are used to perform a masking operation on the image to be dressed up. The mask can accurately mark the area to be dressed up, avoid the influence of the background and other areas, and thus improve the dressing up efficiency.

[0069] Step S104: Extract texture features from the clothing image to obtain the texture features of the target clothing in the clothing image.

[0070] Texture features refer to the visual patterns or texture information of a certain area in an image, including color, pattern, and brightness variations. They can be used to describe the texture of materials. For clothing, texture features reflect information such as the material, pattern, light and shadow, folds, and shape of the clothing.

[0071] In some embodiments, step S104 can be implemented by the following methods: First, the clothing image is standardized, including size adjustment, brightness normalization, etc.; then, the clothing region in the clothing image is identified by a semantic segmentation model or object detection model; next, texture features are extracted from the clothing region. Specifically, any of the following methods can be used for texture feature extraction: Method 1) Filtering method: Use Gabor filter, Laplacian filter, etc. to detect the clothing region to capture texture information of different directions and scales; 2) Feature descriptor: Use local binary mode or scale invariant feature transform, etc. to extract feature points and descriptors of the clothing region, thereby recording the texture features of the clothing region; 3) Deep learning method: Use a pre-trained convolutional neural network to extract deep features, such as VGG or ResNet, and obtain a high-dimensional feature representation of clothing texture through the convolutional layer of the pre-trained convolutional neural network to capture complex texture and pattern information.

[0072] It should be noted that step S104 can capture the texture and details of clothing by extracting the texture features of the clothing image, thereby making the dressing effect more in line with the material of real clothing; moreover, the extracted texture features include detailed information such as clothing patterns and decorations, so that the generated dressing image can retain unique elements such as prints and stripes on the clothing, thereby making the dressing effect more delicate and improving the detail reproduction effect of clothing.

[0073] Step S105: Based on the pose feature map, the masked image to be dressed up, and the texture features of the target clothing, determine the dressing image; the dressing image includes the target object wearing the target clothing.

[0074] Here, feature extraction and feature fusion are performed on the pose feature map and the masked image to be dressed up to obtain the input vector. The input vector and the texture features of the target clothing are then input into the pre-trained main network model to obtain the dressing up image.

[0075] In some embodiments, step S105 can be implemented by the following method: First, feature extraction is performed on the pose feature map to obtain the pose feature vector; then, random Gaussian noise is obtained, and the random Gaussian noise and the masked image to be dressed up are stitched together in the channel dimension, and feature extraction is performed on the stitching result to obtain the stitched image vector. Here, random Gaussian noise is a common image processing noise with Gaussian distribution characteristics. Random Gaussian noise is randomly added to the data with zero mean and adjustable variance to simulate random interference common in nature; next, the stitched image vector and the pose feature vector are added together to obtain the input vector; finally, the input vector and texture features are input into the pre-trained main network model to obtain the dressing image, wherein the dressing image contains the target object wearing the target clothing, such as the dressing image containing the target object A wearing the target clothing B.

[0076] The target object dressing method of this application generates a semantic segmentation map and a pose feature map based on the image to be dressed, which can accurately identify and separate the body regions and pose information of the target object. Moreover, by performing masking processing on the image to be dressed, the interference of the background on the dressing is effectively avoided, so that the dressing effect is focused on the target object itself. Furthermore, by extracting texture features from the clothing image, the texture features of the target clothing are obtained, and the dressing image is generated by combining the texture features, so that the clothing in the dressing image retains the real clothing texture, thereby improving the fidelity and realism of the dressing image, enhancing the visual realism effect, and improving the user experience.

[0077] The following examples illustrate the application scenarios of the target object transformation method provided in this application. This application embodiment can be applied to at least the following exemplary scenarios:

[0078] Scenario 1: With the development of online shopping, when users buy clothes online, in order to determine whether the selected clothes are suitable for them, users can use virtual dress-up technology to replace the selected clothes with their own photos. Thus, the target object dressing method provided in this application embodiment can be adopted. First, the user uploads the original image to be dressed (the user's own photo) and the original clothing image (the photo of the clothing selected by the user) to the terminal. The terminal generates a dressing request based on the received original image to be dressed and the original clothing image, and sends the dressing request to the server. After receiving the dressing request, the server parses the request to obtain the original image to be dressed and the original clothing image, and performs image optimization on the original image to be dressed and the original clothing image to obtain the image to be dressed and the clothing image. Then, it determines the semantic segmentation map of the image to be dressed and the pose feature map of the target object (the user) in the image to be dressed. Next, based on the pose feature map and the semantic segmentation map, it performs masking processing on the image to be dressed to obtain the masked image to be dressed. Then, it extracts texture features from the clothing image to obtain the texture features of the target clothing in the clothing image. Finally, based on the pose feature map, the masked image to be dressed, and the texture features of the target clothing, it determines the dressing image. Through the generated dressing image, the user can determine whether the selected clothes are suitable for them.

[0079] Scenario 2: In the process of online shopping, clothing merchants need to promote their clothing. However, it is too costly for models to try on all the clothes. Therefore, clothing merchants use virtual dress-up technology to replace each piece of clothing in the model photos. Thus, the target object dressing method provided in this application embodiment can be adopted. First, the merchant uploads the original image to be dressed (model photo) and the original clothing image (clothing image from the clothing merchant) to the terminal. The terminal generates a dressing request based on the received original image to be dressed and the original clothing image, and sends the dressing request to the server. After receiving the dressing request, the server parses the request to obtain the original image to be dressed and the original clothing image, and performs image optimization on the original image to be dressed and the original clothing image to obtain the image to be dressed and the clothing image. Then, the semantic segmentation map of the image to be dressed and the pose feature map of the target object (model) in the image to be dressed are determined. Next, based on the pose feature map and the semantic segmentation map, the image to be dressed is masked to obtain the masked image to be dressed. Then, the texture features of the clothing image are extracted to obtain the texture features of the target clothing in the clothing image. Finally, based on the pose feature map, the masked image to be dressed and the texture features of the target clothing, the dressing image is determined. Through the generated dressing image, the merchant can generate promotional images for displaying the clothing.

[0080] The following will use the above scenario one as an example to describe the target object dressing method of this application embodiment. Figure 4This is another optional flowchart illustrating the target object replacement method provided in the embodiments of this application, such as... Figure 4 As shown, the method includes the following steps S201 to S211:

[0081] Step S201: The terminal receives a costume change operation for the original costume image.

[0082] Here, a dress-up application can run on the terminal, and the server constitutes the backend server of the dress-up application. The dress-up operation can be a selection operation or an input operation through the client of the dress-up application running on the terminal. For example, the selection operation can select the original image to be dressed up and the original clothing image, or the input operation can be that the user inputs the original image to be dressed up and the original clothing image on the client.

[0083] In some embodiments, the dress-up application may provide an input interface or input box, allowing users to select or input the original image of the outfit to be changed into and the original image of the clothing. The input interface may be in the form of a form, text box, or drop-down menu, etc., and the specific form is not limited in this application. Users can select the original image of the outfit to be changed into and the original image of the clothing from predetermined options, or manually input the original image of the outfit to be changed into and the original image of the clothing.

[0084] In step S202, the terminal generates a change of clothing request in response to the change of clothing operation.

[0085] Here, the terminal can encapsulate the user-selected original image of the outfit to be changed and the original clothing image into the outfit change request.

[0086] In some embodiments, to ensure the security of dress-up requests, authentication parameters are added to the request. These parameters verify the legitimacy of the request, ensuring that only authorized users can access the dress-up service and preventing unauthorized access and abuse. A common authentication method is to use an API key or token. An API key is a unique string used to identify and verify a user's identity. A token is a credential similar to an access token or authentication token, containing the user's identity information and permissions.

[0087] In step S203, the terminal sends the clothing change request to the server.

[0088] In some embodiments, the terminal sends the encapsulated outfit change request to the server and requests the server to perform the outfit change operation. Typically, the outfit change request is sent using protocols such as HTTP or Web Socket.

[0089] In step S204, the server responds to the dress-changing request by obtaining the original image of the dress to be changed and the original image of the clothing.

[0090] Here, after receiving a clothing change request, the server parses the request. For example, for an HTTP request, the server can parse the request header and request body. Parsing the request header retrieves relevant request information; parsing the request body retrieves the main data of the request, namely the original image of the clothing to be changed into and the original clothing image. Specific fields or parameters in the request body, which contain the original image of the clothing to be changed into and the original clothing image, are parsed. A specific data format, such as JSON or XML, is extracted from the request body, and then this data format is parsed to obtain the original image of the clothing to be changed into and the original clothing image.

[0091] In step S205, the server performs image optimization on the acquired original image to be changed and the original clothing image to obtain the image to be changed and the clothing image. The image to be changed includes the target object.

[0092] The original image to be changed refers to an image containing the target object that has not yet been processed. The original image to be changed usually contains real environmental factors such as natural shooting background and noise, which may affect the clarity of the target object; the original clothing image refers to a clothing image containing the target clothing that has not yet been processed; image optimization refers to the process of preprocessing and enhancing an image, which may include noise reduction, background removal, color correction, etc.

[0093] In some embodiments, in step S205, the server performs image optimization on the acquired original image to be changed into and the original clothing image to obtain the image to be changed into and the clothing image. This can be achieved by the following method: First, acquire the original image to be changed into and the original clothing image; then, preprocess the original image to be changed into and the original clothing image to obtain the preprocessed original image to be changed into and the preprocessed original clothing image; finally, remove the background from the preprocessed original image to be changed into and the preprocessed original clothing image using a pre-trained background removal model to obtain the image to be changed into and the clothing image; the background area in the image to be changed into and the clothing image is transparent or solid color.

[0094] Preprocessing refers to a series of image enhancement and noise removal operations performed on the original images of the clothes to be changed and the original clothing images to improve the image's clarity, contrast, and detail. The background removal model is a deep learning model, usually based on image segmentation technology, used to accurately extract the foreground object or specified subject from the image, remove the background part, and set the background to a transparent color or a solid color.

[0095] In some embodiments, firstly, the user uploads a full-body photo of themselves standing upright (the original image to be changed into) and a photo of a dress (the original clothing image). The full-body photo includes the user's full-body image and a cluttered background taken at home, such as bookshelves, chairs, and lights. The dress photo includes the selected dress and the background shooting environment. Then, the full-body photo and the dress photo are preprocessed. Specifically, for the full-body photo, excess light spots and blurred areas are removed, and the clarity of the user's outline is enhanced. For the dress photo, the dress's color is adjusted to make the color more realistic and vibrant. Next, a pre-trained background removal model (such as MODNet) is used to segment the preprocessed full-body photo and the preprocessed dress photo, generating segmentation masks for the foreground (subject) and background. The edges of the segmentation masks are optimized to reduce blurring or jagged edges. Finally, the generated masks are multiplied by the preprocessed full-body photo and the preprocessed dress photo, respectively, preserving the foreground area and setting the background to a solid color or transparent color to obtain the image to be changed into and the clothing image.

[0096] In some embodiments, the original clothing image includes the target clothing and the original target object wearing the target clothing. Preprocessing the original clothing image to obtain a preprocessed original clothing image can be achieved through the following method: First, the original clothing image is segmented using a pre-trained image segmentation model to obtain a segmentation mask; then, based on the segmentation mask, the background region and human body structure region in the original clothing image are removed to obtain an original clothing image including the target clothing region; finally, feature enhancement is performed on the target clothing region to obtain the preprocessed original clothing image.

[0097] A segmentation mask is a binary image in which target regions (such as clothing) are typically labeled "1" and other regions (such as the human body and background) are typically labeled "0". Segmentation masks are used to distinguish different parts in an image. Human body structure regions refer to the body parts of the original target object, including the head, arms, and legs. Feature enhancement is an operation that optimizes an image, including adjusting the brightness and details of the image.

[0098] In some embodiments, assuming the original clothing image is an image of a model wearing a dress, firstly, the original clothing image is input into a pre-trained image segmentation model. The image segmentation model identifies different regions and generates different region segmentation masks, marking the dress region as the foreground and the model's head, legs, and background as the background. Then, using the generated segmentation mask, the background region is deleted, leaving only the segmented dress region. At this point, the background and the model's body region in the original clothing image have been processed to be transparent, and the original clothing image only contains the outline and color of the dress. Finally, by adjusting parameters such as brightness, contrast, and color saturation, the color of the dress is made more vivid, and the texture details are made more prominent. At the same time, the dress region is sharpened to make the details such as the folds, patterns, and edges of the dress clearer, resulting in the pre-processed original clothing image.

[0099] In step S206, the server determines the semantic segmentation map of the image to be dressed up and the pose feature map of the target object.

[0100] The server uses a pre-trained image segmentation model to perform semantic segmentation on the image to be dressed up, obtaining a semantic segmentation map of the image to be dressed up. At the same time, it uses a pre-trained keypoint detection model to perform keypoint detection on the target object, obtaining a pose feature map of the target object.

[0101] In some embodiments, in step S206, the server determines the semantic segmentation map of the image to be dressed up by the following method: First, the image to be dressed up is preprocessed to obtain a standard image to be dressed up; then, the standard image to be dressed up is segmented by a pre-trained image segmentation model to obtain an image segmentation result containing multiple human body regions and multiple background regions; finally, the pixels in the standard image to be dressed up are classified according to the image segmentation result to obtain the semantic segmentation map of the image to be dressed up.

[0102] A standard image to be changed into clothing refers to an image obtained after preprocessing. Preprocessing includes resizing the image to be changed into clothing so that its size matches the input size of the pre-trained image segmentation model. A semantic segmentation image refers to an image with pixel-level annotations, where each pixel is assigned to a specific semantic category.

[0103] In some embodiments, the size of the image to be changed into clothing is adjusted according to the input requirements of the pre-trained image segmentation model, and the image is scaled or edge-padded to obtain a standard image to change into clothing. Then, the standard image to change into clothing is input into the pre-trained image segmentation model, which identifies multiple regions in the image and generates segmentation results including human body regions, clothing regions, and background regions. Finally, based on the image segmentation results, each pixel in the image to change into clothing is classified, and a category is assigned to each pixel to generate a semantic segmentation map of the image to change into clothing. See also Figure 5 , Figure 5 This is a schematic diagram of determining the semantic segmentation map of the image to be dressed up according to the embodiment of this application. The image to be dressed up 501 is input into the pre-trained image segmentation model 502 to obtain the semantic segmentation map 503 of the image to be dressed up.

[0104] In some embodiments, in step S206, the server determines the pose feature map of the target object by the following method: First, by using a pre-trained keypoint detection model, keypoint detection is performed on the target object in the standard image to be dressed up, obtaining the coordinates of multiple keypoints of the target object and the keypoint confidence corresponding to each keypoint coordinate; then, the keypoint coordinates are filtered based on the keypoint confidence to obtain the target keypoint coordinates; finally, based on the target keypoint coordinates, the target keypoints corresponding to the target keypoint coordinates are marked on the standard image to be dressed up, thus obtaining the pose feature map of the target object.

[0105] Keypoint detection models are deep learning models that can automatically identify various joints of the human body. They are typically pre-trained on large datasets of images containing human keypoints to efficiently detect and label the human skeleton or pose in images. Keypoint coordinates refer to the two-dimensional coordinates of each keypoint (such as knuckles, shoulders, elbows, etc.) in the image, usually in (x, y) format. Keypoint confidence refers to the reliability of the keypoint detection model's prediction of each keypoint coordinate; a higher value indicates higher accuracy of the coordinate.

[0106] In some embodiments, a standard image of the subject to be dressed up is input into a pre-trained keypoint detection model, such as OpenPose or MediaPipe. The pre-trained keypoint detection model automatically identifies key points of the target object's body, such as the head, shoulders, elbows, and wrists. Then, the pre-trained keypoint detection model outputs the coordinates and corresponding confidence scores of each keypoint. For example, the coordinates of the right shoulder are (100, 250), the coordinates of the left shoulder are (120, 260), the confidence score of the right shoulder keypoint is 0.9, and the confidence score of the left shoulder keypoint is 0.92. The pre-trained keypoint model can then adjust the coordinates based on the confidence scores and preset values ​​of each keypoint. A confidence threshold is used to filter the coordinates of multiple keypoints generated by the model. For example, if the preset keypoint threshold is 0.8, and the confidence scores of the keypoints for the left and right shoulders are both greater than 0.8, then these two keypoints will be retained for subsequent processing. Conversely, if the confidence score of a keypoint for a certain joint is 0.5, then that point will be deleted to reduce the impact of unreliable keypoints on subsequent processing. Finally, based on the selected target keypoint coordinates, they are marked on the standard image to be dressed. Each target keypoint can be represented by a small dot, and adjacent small dots are connected by line segments to form the pose skeleton of the target object, thus obtaining the pose feature map of the target object. See also Figure 6 , Figure 6 This is a schematic diagram of the pose feature map of the target object provided in the embodiment of this application. The image 601 to be dressed up is input into the pre-trained key point detection model 602 to obtain the pose feature map 603 of the image to be dressed up.

[0107] In step S207, the server performs masking processing on the image to be dressed up based on the pose feature map and semantic segmentation map to obtain the masked image to be dressed up.

[0108] The server performs masking processing on the image to be dressed up based on the pose feature map and semantic segmentation map, resulting in a masked image of the image to be dressed up, with the dressing area represented by a mask. This masked image can more accurately separate the dressing area from the background area. See also... Figure 7 , Figure 7 This is a schematic diagram of the masking process performed on the image to be dressed up based on the pose feature map and semantic segmentation map provided in this application embodiment, to obtain the masked image to be dressed up. The masking process is performed on the image to be dressed up based on the pose feature map 701 and semantic segmentation map 702 to obtain the masked image to be dressed up 703.

[0109] In some embodiments, the semantic segmentation map includes label information for each region in the image to be dressed up; wherein, the regions include human body regions and background regions; see also Figure 8 , Figure 8 This is a flowchart illustrating the image to be dressed up after the mask is determined, as provided in an embodiment of this application. Figure 8The following step S207 illustrates how the server performs masking processing on the image to be dressed up based on the pose feature map and semantic segmentation map, obtaining the masked image to be dressed up. This can be achieved through steps S2071 to S2075:

[0110] In step S2071, the server determines the target region from the semantic segmentation map based on the label information and generates a binary image of the target region.

[0111] Label information refers to the specific category label assigned to each region in the semantic segmentation map to represent the specific region in the image, such as the head region, upper limb region, and leg region; the target region refers to the region in the standard image to be dressed up that needs to be dressed up; a binary image is an image that contains only black and white colors. Binary images are often used to distinguish between target regions and background regions, and are often used in mask generation or background removal tasks. In this case, white regions represent target regions, and black regions represent non-target regions.

[0112] In some embodiments, firstly, the server analyzes the semantic segmentation map based on the tag information to determine the pixel position corresponding to the "shirt" tag; then, a binary image is generated based on the determined pixel position, wherein in the generated binary image, the pixel labeled "shirt" is set to white (value 1), and the rest is set to black (value 0); it should be noted that if there are multiple parts that need to be changed, such as changing the shirt and pants, the pixel labeled "shirt" is set to white (value 1), the pixel labeled "pants" is set to green (value 2), and the rest is set to black (value 0). Here, the color corresponding to the value of each pixel can be set by the user, and this application does not limit it.

[0113] In step S2072, the server adjusts the contour of the binary image based on the coordinates of the target key points in the pose feature map to obtain the adjusted binary image.

[0114] In some embodiments, the server adjusts the contour of the binary image based on the keypoint coordinates in the pose feature map. This can be achieved through the following method: First, the server checks whether the contour of the binary image covers all keypoint coordinates. If the keypoint coordinates exceed or deviate from the contour, the server adjusts the boundary of the binary image based on the target keypoint coordinates to ensure that the binary image contains all target keypoints. Then, based on the distribution of the target keypoints, the server redraws the edges of the binary image to make the edges smoother and conform to the shape of the target object. Finally, when adjusting the contour, image morphological operations, such as dilation or erosion, can be used to make the contour fit the target object more closely.

[0115] In step S2073, the server performs smoothing processing on the adjusted binary image to obtain an optimized binary image.

[0116] Smoothing is an image processing technique used to reduce noise or edge roughness in an image and make the image edges smoother. Smoothing methods can include Gaussian blur, mean filtering, etc., which are not limited to this application.

[0117] In some embodiments, Gaussian blur can be used to smooth the adjusted binary image. First, a 3×3 Gaussian kernel is selected and applied to the edge region of the binary image. Then, a Gaussian weighted average of each pixel is calculated, and the result is used to replace the original pixel value, thereby obtaining a binary image with softer edges.

[0118] In step S2074, the server aligns the optimized binary image with the standard image to be dressed up, resulting in an aligned image.

[0119] Image alignment refers to the precise superposition of two images in spatial position so that the outline or region of the target object in the optimized binary image coincides with that in the standard image to be dressed.

[0120] In some embodiments, important feature points, such as human body contours, are extracted from the standard image to be dressed and the optimized binary image. Then, corresponding matching point pairs are found based on the important feature points in the standard image to be dressed and the optimized binary image. Next, geometric transformation parameters, such as translation and rotation angles, are calculated using the matching point pairs. Finally, the optimized binary image is transformed so that the optimized binary image and the area to be dressed in the standard image to be dressed precisely coincide, resulting in an aligned image.

[0121] In step S2075, the server, based on the aligned image, replaces the pixel values ​​in the occluded area corresponding to the optimized binary image with the corresponding area in the standard image to be dressed up, thus obtaining the masked image to be dressed up.

[0122] In some embodiments, the occluded area is the area marked as the clothing to be changed in the binary image; the server traverses the occluded area of ​​the optimized binary image pixel by pixel, and copies the pixel value of each pixel in the occluded area to the corresponding pixel in the standard clothing to be changed image, so as to obtain the masked clothing to be changed image.

[0123] In step S208, the server extracts texture features from the clothing image to obtain the texture features of the target clothing in the clothing image.

[0124] Texture features refer to the characteristics of clothing such as material, folds, and patterns, and can also include the shape of the clothing and the natural light and shadow within it.

[0125] In some embodiments, step S208 can be implemented by the following method: First, global texture features are extracted from the clothing image using a pre-trained image encoder to obtain the global texture features of the target clothing in the clothing image; then, local texture features are extracted from the clothing image using a pre-trained texture feature extraction model to obtain the local texture features of the target clothing; finally, the global texture features and local texture features are determined as the texture features of the target clothing.

[0126] An image encoder is a neural network model that is typically used to convert images into feature vectors. Global texture features describe the overall texture style and structural information of clothing, which can include information such as the material and shape of the clothing. Local texture features focus on local areas of the clothing image, such as clothing folds, lighting information, and detailed textures. Local texture information is used to capture the texture and texture details of specific parts of the clothing.

[0127] In some embodiments, for a cotton striped shirt, firstly, a pre-trained image encoder is used to obtain global texture features, which include the overall material information of the shirt, such as color tone and the repeating pattern of the stripes. Then, a pre-trained texture feature extraction model is used to detect and identify local regions of the image, obtaining the detailed texture of each stripe edge and shirt fold, ensuring that local texture details (such as line thickness and fold position) are preserved. Finally, the global and local texture features are determined as the texture features of the target garment. By extracting the texture features of the target garment, the final dressing image presents the realistic details of the clothing.

[0128] In step S209, the server determines the dressing image based on the pose feature map, the masked image to be dressed, and the texture features of the target clothing; the dressing image includes the target object wearing the target clothing.

[0129] The server inputs the pose feature map, the masked image to be dressed up, and the texture features of the target clothing into the pre-trained main network model to obtain the dressing up image, which includes the target object wearing the target clothing. If the image to be dressed up contains target object A and the clothing image contains target clothing B, then the final generated dressing up image will include target object A wearing target clothing B.

[0130] In some embodiments, see Figure 9 , Figure 9 This is a schematic diagram of the process of determining the dressing image based on the pose feature map, the masked image to be dressed, and the texture features of the target garment, provided in the embodiments of this application; Figure 9The diagram shows that in step S209, the server determines the dressing image based on the pose feature map, the masked image to be dressed, and the texture features of the target clothing; the dressing image includes the target object wearing the target clothing, which can be achieved through the following steps S2091 to S2095:

[0131] In step S2091, the server extracts features from the pose feature map using a pre-trained pose encoder to obtain a pose feature vector.

[0132] A pose encoder is a deep learning model used to encode an input pose feature map into a pose feature vector. The pose feature vector is a numerical representation of the pose feature map of a target object. The pose feature vector contains key information about the pose, such as the relative positions of body parts and the relationships between joints.

[0133] Step S2092: The server obtains random Gaussian noise.

[0134] Random Gaussian noise is a type of random noise that follows a Gaussian distribution. Adding random Gaussian noise to an image can simulate interference under natural shooting conditions and is often used for data augmentation. Random Gaussian noise is usually determined by two parameters: mean (representing the central value of the noise intensity) and variance (controlling the degree of noise diffusion).

[0135] In step S2093, the server stitches the random Gaussian noise and masked image to be dressed up in the channel dimension to generate a stitched image tensor.

[0136] Channel dimension is often used to store color information. For example, an RGB image contains three channels (red, blue, and green). An image tensor is a multidimensional array that represents image data. The dimensions of a tensor represent the structural information of the image, such as width, height, and color channels.

[0137] In some embodiments, step S2093 can be implemented by the following method: First, since the acquired random Gaussian noise does not have a channel dimension, it is necessary to expand the random Gaussian noise into a single channel (e.g., with a shape of H×X×1) so that the random Gaussian noise can be aligned with the masked image to be dressed up in the channel dimension; then, the expanded random Gaussian noise and the masked image to be dressed up are concatenated in the channel dimension. For example, if the masked image to be dressed up is a three-channel RGB image with a shape of H×X×3, after concatenating the expanded random Gaussian noise and the masked image to be dressed up in the channel dimension, a concatenated image tensor containing four channels is obtained, and the shape of the concatenated image tensor is H×X×4.

[0138] In step S2094, the server extracts features from the stitched image tensor to obtain the stitched image vector.

[0139] The server extracts features from the stitched image tensor using a pre-trained feature extraction model to obtain a stitched image vector, where the stitched image vector is a numerical representation of the stitched image tensor.

[0140] In step S2095, the server determines the dressing image based on the stitched image vector, pose feature vector, and texture features of the target clothing.

[0141] The server inputs the stitched image vector, pose feature vector, and texture features of the target clothing into the pre-trained main network model to obtain the final altered image.

[0142] In some embodiments, the texture features of the target garment include global texture features and local texture features; step S2095 can be implemented by the following method: first, the stitched image vector and the pose feature vector are added together to obtain the input vector; then, the input vector, global texture features and local texture features are input into the pre-trained main network model to obtain the dressing image; wherein, the pre-trained main network model includes multiple data processing modules connected in sequence; the input vector is input into the first data processing layer of the first data processing module in the pre-trained main network model, the global texture features are input into the cross-attention layer of each data processing module in the pre-trained main network model, and the local texture information is input into the self-attention mechanism layer of each data processing module in the pre-trained main network model.

[0143] The input vector is obtained by adding the stitched image vector and the pose feature vector. The input vector is the input to the pre-trained main network model. The main network model is a deep neural network model, which includes multiple data processing models. Each data processing model contains different data processing layers. The main network model is used to generate the final transformation image. The cross-attention mechanism is used to focus on the relationship between different feature sets. By associating input features and target features, it understands and fuses the correlation between individual features. The self-attention mechanism focuses on the correlation within the same feature set. By calculating the relationship within features, the self-attention mechanism can capture the detailed correlation between local features.

[0144] In some embodiments, see Figure 10 , Figure 10 This is an input diagram of the first data processing module in the pre-trained main network model provided in this application embodiment. The input vector is obtained by adding the spliced ​​image vector and the pose feature vector, and is input into the layer normalization 1001 (first data processing layer) in the first data processing module; the global texture features are input into the cross attention mechanism layer 1003; and the local texture features are input into the self attention mechanism layer 1002.

[0145] In some embodiments, the original costume-changing network can also be trained, wherein the original costume-changing network includes an original texture feature extraction model, an original image encoder, an original pose encoder, and an original main network model. The training process of the original costume-changing network can be implemented through the following steps:

[0146] 1) Obtain the training dataset and a pre-trained text-to-image diffusion model, such as the stablediffusion-v1.5 model, and modify the pre-trained text-to-image generation model as follows: Figure 11 The dual-branch structure shown uses the modified dual-branch network as the original replacement network; the parameters of the original texture feature extraction model (Reference UNet) and the original main network model (MainUNet) are both derived from the pre-trained text-to-image model.

[0147] 2) Initialize the original pose encoder, which is a 4-layer convolutional neural network used to convert the pose feature map into a pose feature vector.

[0148] 3) Input the clothing images to be trained in the training dataset into the original image encoder (CLIP ImageEncoder) to extract the global texture features of the clothing in the clothing images to be trained.

[0149] 4) Input the clothing images to be trained in the training dataset into the original texture feature extraction model to extract the local texture features of the clothing in the clothing images to be trained.

[0150] 5) Concatenate the occluded images in the training dataset with a random Gaussian noise along the channel dimension to obtain a concatenated image vector; then, extract features from the pose feature maps in the training dataset using the original pose encoder to obtain a pose feature vector; finally, add the concatenated image vector and the pose feature vector together as the input vector of the original main network model.

[0151] 6) Input the input vector, global texture features, and local texture features into the original main network model. During the forward propagation of the original main network model, the input of the self-attention mechanism layer of each data processing module is updated according to formula (1), and the features h of the original main network model are updated accordingly. m Features h of the original texture feature extraction model c To combine.

[0152] h = concat(h) m ,h c ) Formula (1)

[0153] Where h represents the combined features of the original main network model and the original texture feature extraction model; concat represents the addition operation; h mRepresents the characteristics of the original master network model; h c This represents the features of the original texture feature extraction model.

[0154] 7) Use the difference between the output prediction noise of the original main network model and the random Gaussian noise as the loss function to optimize the parameters of the original replacement network.

[0155] 8) Repeat steps 2-7 on the training dataset until the original costume network converges to obtain the pre-trained costume network.

[0156] In step S210, the server sends the costume change image to the terminal.

[0157] Step S211: The terminal displays the costume change image on the current interface.

[0158] After receiving the outfit images, users can determine whether the selected clothes suit them based on the images.

[0159] The target object dressing method of this application generates a semantic segmentation map and a pose feature map based on the image to be dressed, which can accurately identify and separate the body regions and pose information of the target object. Moreover, by performing masking processing on the image to be dressed, the interference of the background on the dressing is effectively avoided, so that the dressing effect is focused on the target object itself. Furthermore, by extracting texture features from the clothing image, the texture features of the target clothing are obtained, and the dressing image is generated by combining the texture features, so that the clothing in the dressing image retains the real clothing texture, thereby improving the fidelity and realism of the dressing image, enhancing the visual realism effect, and improving the user experience.

[0160] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0161] This application provides a target object dressing method based on a diffusion model. Through a two-branch network structure, a clothing image (the original clothing image) and a model image (the original image to be dressed) are input respectively, and a dressing image of the model wearing the clothing can be generated.

[0162] The target object dressing method provided in this application embodiment will be described below from three aspects: data preparation, model training, and model inference.

[0163] First aspect: Data preparation.

[0164] 1) Prepare a batch of flat lay images of clothing (I_cloth) and corresponding images of models wearing the clothing (I_model).

[0165] 2) Use a pre-trained keypoint detection model, such as the OpenPose model, to perform keypoint detection on the model image I_model and obtain the keypoint information I_pose of the model in the model image.

[0166] 3) Using a pre-trained image segmentation model, such as the CIHP_PGN model, perform image segmentation on the model image I_model to obtain segmentation images I_parsing for different regions of the human body.

[0167] 4) Combine the information from I_pose and I_parsing to perform a masking operation on the clothing area of ​​the model image I_model to obtain the masked model image I_agn.

[0168] The training dataset can be constructed by performing steps 2-4 above on each pair of images in the original dataset.

[0169] The second aspect: model training.

[0170] 1) Modify the pre-trained text-based image diffusion model, such as the stablediffusion-v1.5 model, to... Figure 11 The dual-branch structure shown uses the modified dual-branch network as the original replacement network; the parameters of the original texture feature extraction model (Reference UNet) and the original main network model (Main UNet) are both derived from the pre-trained text-to-image model.

[0171] 2) Initialize a raw pose encoder, wherein the pose encoder is a 4-layer convolutional neural network used to convert key point information (the pose feature map above) into feature information (the pose feature vector above).

[0172] 3) Input the clothing images to be trained in the training dataset into the original image encoder (CLIP ImageEncoder) to extract the global texture features of the clothing in the clothing images to be trained.

[0173] 4) Input the clothing images to be trained in the training dataset into the original texture feature extraction model to extract the local texture features of the clothing in the clothing images to be trained.

[0174] 5) Concatenate the occluded images in the training dataset with a random Gaussian noise I_noise along the channel dimension to obtain a concatenated image vector; then, extract features from the pose feature maps in the training dataset using the original pose encoder to obtain a pose feature vector; finally, add the concatenated image vector and the pose feature vector together as the input vector of the original main network model.

[0175] 6) Input the input vector, global texture features, and local texture features into the original main network model. During the forward propagation of the original main network model, the input of the self-attention mechanism layer of each data processing module is updated according to the above formula (1), and the features h of the original main network model are updated. m Features h of the original texture feature extraction model c To combine.

[0176] 7) Use the difference between the output prediction noise of the original main network model and the random Gaussian noise as the loss function to optimize the parameters of the original replacement network.

[0177] 8) Repeat steps 2-7 on the training dataset until the original costume network converges to obtain the pre-trained costume network.

[0178] The third aspect: model reasoning.

[0179] 1) Prepare an image of the outfit to be changed and an image of the outfit as input for the outfit changing network.

[0180] 2) Determine the semantic segmentation map of the image to be dressed up, and the pose feature map of the target objects included in the image to be dressed up.

[0181] 3) Based on the pose feature map and semantic segmentation map, determine the mask of the clothing region and apply it to the image to be dressed, to obtain the masked image to be dressed.

[0182] 4) Input the clothing image into the pre-trained texture feature extraction model to extract the local texture features of the target clothing, and input the clothing image into the pre-trained image encoder to extract the global texture features of the target clothing.

[0183] 5) The masked image to be dressed up is concatenated with a random Gaussian noise along the channel dimension to obtain a concatenated vector; then, the pose feature map is extracted by a pre-trained pose encoder to obtain a pose feature vector; finally, the concatenated vector and the pose feature vector are added together as the input vector of the pre-trained main network model.

[0184] 6) Input the input vector, global texture features, and local texture features into the pre-trained main network model. During the forward propagation of the pre-trained main network model, the input of the self-attention mechanism layer of each data processing module is updated according to the above formula (1). The features h of the pre-trained main network model are then used to update the input of the self-attention mechanism layer of each data processing module. m Features h of the pre-trained texture feature extraction model c To combine.

[0185] 7) Apply the predicted noise from the output of the pre-trained main network model to random Gaussian noise to obtain the denoised image at the current time.

[0186] 8) Repeat steps 5-8 several times to obtain the final denoised image, which is the dress-up image.

[0187] The target object replacement method provided in this application embodiment is described in [reference]. Figure 12 , Figure 12 This is a rendering of the changing image provided in the embodiments of this application; by extracting the texture features of the target garment and combining them with the texture features of the garment to generate the changing image, the garment in the changing image retains the real garment texture, thereby improving the fidelity and realism of the changing image, and can better fit the body shape of different models.

[0188] It is understood that in the embodiments of this application, if data related to user information or enterprise information is involved, when the embodiments of this application are applied to specific products or technologies, it is necessary to obtain user permission or consent, or to obfuscate this information in order to eliminate the correspondence between this information and the user; and the collection and processing of related data should strictly comply with the requirements of relevant laws and regulations when applied in practice, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0189] The following continues to describe an exemplary structure of the target object changing device 254 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the target object dressing device 254 includes: an acquisition module 2541, used to acquire an image to be dressed and a clothing image; the image to be dressed includes a target object; a first determination module 2542, used to determine a semantic segmentation map of the image to be dressed and a pose feature map of the target object; a masking module 2543, used to perform masking processing on the image to be dressed based on the pose feature map and the semantic segmentation map to obtain a masked image to be dressed; a feature extraction module 2544, used to extract texture features from the clothing image to obtain the texture features of the target clothing in the clothing image; and a second determination module 2545, used to determine a dressing image based on the pose feature map, the masked image to be dressed, and the texture features of the target clothing; the dressing image includes a target object wearing the target clothing.

[0190] In some embodiments, the first determining module is further configured to: preprocess the image to be changed into to obtain a standard image to be changed into; perform image segmentation on the standard image to be changed into using a pre-trained image segmentation model to obtain an image segmentation result containing multiple human body regions and multiple background regions; and classify the pixels in the standard image to be changed into according to the image segmentation result to obtain a semantic segmentation map of the image to be changed into.

[0191] In some embodiments, the first determining module is further configured to: perform keypoint detection on the target object in the standard image to be dressed up using a pre-trained keypoint detection model, to obtain multiple keypoint coordinates of the target object and keypoint confidence corresponding to each keypoint coordinate; filter the keypoint coordinates based on the keypoint confidence to obtain target keypoint coordinates; and mark the target keypoints corresponding to the target keypoint coordinates on the standard image to be dressed up based on the target keypoint coordinates to obtain the pose feature map of the target object.

[0192] In some embodiments, the semantic segmentation map includes label information for each region in the image to be dressed up; the region includes the human body region and the background region; the masking module is further configured to: determine a target region from the semantic segmentation map based on the label information, and generate a binary image of the target region; adjust the contour of the binary image based on the key point coordinates in the pose feature map to obtain an adjusted binary image; smooth the adjusted binary image to obtain an optimized binary image; align the optimized binary image with the standard image to be dressed up to obtain an aligned image; and based on the aligned image, replace the pixel values ​​in the occluded region corresponding to the optimized binary image with the region corresponding to the occluded region in the standard image to be dressed up to obtain the masked image to be dressed up.

[0193] In some embodiments, the feature extraction module is further configured to: extract global texture features from the clothing image using a pre-trained image encoder to obtain global texture features of the target clothing in the clothing image; extract local texture features from the clothing image using a pre-trained texture feature extraction model to obtain local texture features of the target clothing; and determine the global texture features and the local texture features as the texture features of the target clothing.

[0194] In some embodiments, the second determining module is further configured to: extract features from the pose feature map using a pre-trained pose encoder to obtain a pose feature vector; acquire random Gaussian noise; stitch the random Gaussian noise and the masked image to be changed in the channel dimension to generate a stitched image tensor; extract features from the stitched image tensor to obtain a stitched image vector; and determine the changing image based on the stitched image vector, the pose feature vector, and the texture features of the target clothing.

[0195] In some embodiments, the texture features of the target clothing include global texture features and local texture features; the second determining module is further configured to: add the stitched image vector and the pose feature vector to obtain an input vector; input the input vector, the global texture features and the local texture features into a pre-trained main network model to obtain a dressing image; wherein, the pre-trained main network model includes multiple data processing modules connected in sequence; the input vector is input into the first data processing layer of the first data processing module in the pre-trained main network model, the global texture features are input into the cross-attention layer of each data processing module in the pre-trained main network model, and the local texture information is input into the self-attention mechanism layer of each data processing module in the pre-trained main network model.

[0196] In some embodiments, the acquisition module is further configured to: acquire an original image of the outfit to be changed into and an original image of the clothing; preprocess the original image of the outfit to be changed into and the original image of the clothing to obtain a preprocessed original image of the outfit to be changed into and a preprocessed original image of the clothing to be changed into; remove the background from the preprocessed original image of the outfit to be changed into and the preprocessed original image of the clothing to be changed into using a pre-trained background removal model to obtain the image of the outfit to be changed into and the image of the clothing to be changed into; the background area in the image of the outfit to be changed into and the image of the clothing to be changed into is transparent or solid color.

[0197] In some embodiments, the original clothing image includes the target clothing and the original target object wearing the target clothing; the apparatus further includes a processing module for performing image segmentation on the original clothing image using a pre-trained image segmentation model to obtain a segmentation mask; based on the segmentation mask, deleting the background region and human body structure region in the original clothing image to obtain an original clothing image including the target clothing region; and performing feature enhancement on the target clothing region to obtain the preprocessed original clothing image.

[0198] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0199] This application provides an electronic device, including: a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to implement the above-described target object replacement method.

[0200] This application provides a computer program product, which includes computer-executable instructions or a computer program stored in a computer-readable storage medium; wherein, when the processor of an electronic device reads the computer-executable instructions or the computer program from the computer-readable storage medium and executes the computer-executable instructions or the computer program, the above-mentioned target object replacement method is implemented.

[0201] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will perform the aforementioned target object replacement method, for example, such as... Figure 3 The method shown.

[0202] In some embodiments, the storage medium may be a computer-readable storage medium, such as a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEROM), flash memory, magnetic surface memory, optical disk, or a compact disk-read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0203] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0204] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file containing other programs or data, for example, in one or more scripts within a Hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0205] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for changing the appearance of a target object, characterized in that, The method includes: Acquire the image of the outfit to be changed and the image of the clothing; the image of the outfit to be changed includes the target object; Determine the semantic segmentation map of the image to be dressed up and the pose feature map of the target object; Based on the pose feature map and the semantic segmentation map, the image to be dressed up is masked to obtain the masked image to be dressed up. Texture features are extracted from the clothing image to obtain the texture features of the target clothing in the clothing image; Based on the pose feature map, the masked image to be dressed up, and the texture features of the target clothing, a dressing up image is determined; the dressing up image includes a target object wearing the target clothing.

2. The method according to claim 1, characterized in that, Determining the semantic segmentation map of the image to be dressed up includes: The image to be changed into clothing is preprocessed to obtain a standard image to be changed into clothing; The standard image to be changed into clothing is segmented using a pre-trained image segmentation model to obtain image segmentation results containing multiple human body regions and multiple background regions. Based on the image segmentation results, the pixels in the standard image to be dressed up are classified to obtain the semantic segmentation map of the image to be dressed up.

3. The method according to claim 2, characterized in that, Determining the pose feature map of the target object includes: By using a pre-trained keypoint detection model, keypoint detection is performed on the target object in the standard image to be dressed up, and multiple keypoint coordinates of the target object and the keypoint confidence corresponding to each keypoint coordinate are obtained. The key point coordinates are filtered based on the key point confidence level to obtain the target key point coordinates; Based on the coordinates of the target key points, the target key points corresponding to the coordinates of the target key points are marked on the standard image to be dressed, and the posture feature map of the target object is obtained.

4. The method according to claim 3, characterized in that, The semantic segmentation map includes label information for each region in the image to be dressed up; the region includes the human body region and the background region. The process of masking the image to be dressed up based on the pose feature map and the semantic segmentation map to obtain the masked image to be dressed up includes: Based on the label information, the target region is determined from the semantic segmentation map, and a binary image of the target region is generated; Based on the coordinates of the target key points in the pose feature map, the contour of the binary image is adjusted to obtain the adjusted binary image. The adjusted binary image is smoothed to obtain an optimized binary image; The optimized binary image is aligned with the standard image to be changed into clothing to obtain an aligned image; Based on the aligned image, the pixel values ​​in the occlusion region corresponding to the optimized binary image are replaced in the region corresponding to the occlusion region in the standard image to be dressed up, to obtain the masked image to be dressed up.

5. The method according to claim 1, characterized in that, The step of extracting texture features from the clothing image to obtain the texture features of the target clothing in the clothing image includes: The global texture features of the target garment in the clothing image are extracted by a pre-trained image encoder. By using a pre-trained texture feature extraction model, local texture features are extracted from the clothing image to obtain the local texture features of the target clothing. The global texture features and the local texture features are determined as the texture features of the target garment.

6. The method according to claim 1, characterized in that, The step of determining the dressing image based on the pose feature map, the masked image to be dressed, and the texture features of the target garment includes: The pose feature map is used to extract features by a pre-trained pose encoder to obtain a pose feature vector. Obtain random Gaussian noise; The random Gaussian noise and the masked image to be dressed are stitched together in the channel dimension to generate a stitched image tensor; Feature extraction is performed on the stitched image tensor to obtain the stitched image vector; The dressing image is determined based on the stitched image vector, the pose feature vector, and the texture features of the target clothing.

7. The method according to claim 6, characterized in that, The texture features of the target garment include global texture features and local texture features; Determining the dressing image based on the stitched image vector, the pose feature vector, and the texture features of the target clothing includes: The stitched image vector and the pose feature vector are added together to obtain the input vector; The input vector, the global texture features, and the local texture features are input into the pre-trained main network model to obtain the costume change image; The pre-trained main network model includes multiple data processing modules connected in sequence; the input vector is input to the first data processing layer of the first data processing module in the pre-trained main network model; the global texture features are input to the cross-attention layer of each data processing module in the pre-trained main network model; and the local texture information is input to the self-attention mechanism layer of each data processing module in the pre-trained main network model.

8. The method according to claim 1, characterized in that, The acquisition of the image of the outfit to be changed and the image of the clothing includes: Obtain the original image of the outfit to be changed and the original image of the clothing; The original image to be changed into and the original clothing image are preprocessed to obtain the preprocessed original image to be changed into and the preprocessed original clothing image. The background is removed from the pre-trained original image to be changed into clothing and the pre-processed original clothing image by using a pre-trained background removal model to obtain the image to be changed into clothing and the clothing image; the background area in the image to be changed into clothing and the clothing image is transparent or solid color.

9. The method according to claim 8, characterized in that, The original clothing image includes the target clothing and the original target object wearing the target clothing; The original clothing image is preprocessed to obtain a preprocessed original clothing image, including: The original clothing image is segmented using a pre-trained image segmentation model to obtain a segmentation mask. Based on the segmentation mask, the background region and human body structure region in the original clothing image are deleted to obtain the original clothing image including the target clothing region. Feature enhancement is performed on the target clothing area to obtain the preprocessed original clothing image.

10. A target object changing device, characterized in that, The device includes: The acquisition module is used to acquire the image of the outfit to be changed into and the image of the clothing; the image of the outfit to be changed into includes the target object; The first determining module is used to determine the semantic segmentation map of the image to be changed and the pose feature map of the target object; The masking module is used to perform masking processing on the image to be dressed up based on the pose feature map and the semantic segmentation map to obtain the masked image to be dressed up. The feature extraction module is used to extract texture features from the clothing image to obtain the texture features of the target clothing in the clothing image; The second determining module is used to determine a changing image based on the posture feature map, the masked image to be changed, and the texture features of the target clothing; the changing image includes a target object wearing the target clothing.

11. An electronic device, characterized in that, include: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the target object replacement method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The device stores computer-executable instructions or computer programs for causing a processor to execute the computer-executable instructions or computer programs to implement the target object replacement method according to any one of claims 1 to 9.

13. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the target object replacement method according to any one of claims 1 to 9 is implemented.

Images