Image processing model training method and device, image processing method and device
Through the image processing model training method, the reference network and backbone network are used to integrate noise characteristics, and the clothing and background information in the virtual fitting are adaptively learned, which solves the generation effect problem caused by the uncertainty of mask information in virtual fitting technology, and realizes the Higher quality image generation.
Patent Information
- Application Number
- CN202411103700.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-12
AI Technical Summary
In the existing virtual fitting technology, the image completion method based on diffusion model relies on accurate mask information. The mask quality uncertainty will seriously affect the image generation effect and it is difficult to accurately control the generation of clothing and background elements.
Through the image processing model training method, the reference network and backbone network are used to perform feature processing on the target object and simulated object, merge noise characteristics, and adaptively learn the posture and clothing information of the simulated object, avoiding the use of mask information and realizing the clothing and background Accurate control.
It improves the quality and effect of image generation, can better retain high-level semantic information such as clothing styles and materials, adapt to more complex image processing scenarios, and reduces the manual distinction process.
Smart Images

Figure CN118644586B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this specification relate to the field of computer technology, and in particular to an image processing model training method and device, an image processing method and device method. Background Art
[0002] With the rapid development of e-commerce, virtual fitting technology has become an important part of the online shopping experience; however, existing virtual fitting technology still has some problems, such as the difficulty in accurately controlling the generation of clothing and background elements, and poor performance on clothing of different styles and designs.
[0003] Among the existing methods, a common way to implement the virtual dressing task is to complete the image based on the diffusion model. However, the method of completing the image based on the diffusion model usually requires an accurate mask (a mask is used to indicate the part of the image that needs to be filled, and the diffusion model is generated in these masked areas) to indicate the image processing area to be processed, that is, the dressing area, and image completion is performed in this dressing area to achieve the dressing effect. However, if there is a deviation in the mask area, it will seriously affect the generation effect of the image generated by the diffusion model. Summary of the invention
[0004] In view of this, an embodiment of this specification provides an image processing model training method. One or more embodiments of this specification also relate to an image processing model training device, an image processing method, an image processing device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art that when generating images based on image completion, the image generation effect is seriously affected by the provided mask information due to the uncertainty of the mask quality.
[0005] According to a first aspect of an embodiment of this specification, there is provided an image processing model training method, comprising:
[0006] Determine a target object image, a simulated object image, a posture image of a simulated object in the simulated object image, a target simulated object image, and random noise, wherein a reference object of the simulated object in the simulated object image is different from a target object in the target object image, and the target simulated object image is a simulated object image in which the reference object is replaced by the target object;
[0007] Using a reference network of an image processing model, feature processing is performed on the target object image and the simulated object image respectively to obtain target object reference features and simulated object reference features, and the target object reference features and the simulated object reference features are input into a backbone network of the image processing model;
[0008] Using the backbone network of the image processing model, feature processing is performed on the target object reference feature, the simulated object reference feature and the fused noise feature to obtain a predicted simulated object image, wherein the fused noise feature is obtained by fusing the posture feature of the posture image with the random noise;
[0009] The image processing model is trained according to the predicted simulated object image and the target simulated object image to obtain a trained image processing model.
[0010] According to a second aspect of an embodiment of this specification, there is provided an image processing model training device, comprising:
[0011] an image determination module, configured to determine a target object image, a simulated object image, a posture image of a simulated object in the simulated object image, a target simulated object image, and random noise, wherein a reference object of the simulated object in the simulated object image is different from a target object in the target object image, and the target simulated object image is a simulated object image in which the reference object is replaced by the target object;
[0012] a feature processing module configured to perform feature processing on the target object image and the simulated object image respectively using a reference network of the image processing model, obtain target object reference features and simulated object reference features, and input the target object reference features and the simulated object reference features into a backbone network of the image processing model;
[0013] An image acquisition module is configured to perform feature processing on the target object reference feature, the simulated object reference feature and the fused noise feature using the backbone network of the image processing model to obtain a predicted simulated object image, wherein the fused noise feature is obtained by fusing the posture feature of the posture image with the random noise;
[0014] The model training module is configured to train the image processing model according to the predicted simulated object image and the target simulated object image to obtain a trained image processing model.
[0015] According to a third aspect of the embodiments of this specification, there is provided an image processing method, including:
[0016] Determine the image of the model to be changed and the image of the clothing to be replaced;
[0017] The image of the model to be changed and the image of the clothing to be replaced are input into an image processing model, and the clothing in the image of the model to be changed is replaced with the clothing in the image of the clothing to be replaced using the image processing model to obtain a target image of the model to be changed, wherein the image processing model is obtained by the above-mentioned image processing model training method.
[0018] According to a fourth aspect of the embodiments of this specification, there is provided an image processing device, including:
[0019] An image determination module is configured to determine an image of a model to be changed and an image of clothing to be replaced;
[0020] The image acquisition module is configured to input the image of the model to be changed and the image of the clothing to be replaced into an image processing model, and use the image processing model to replace the clothing in the image of the model to be changed with the clothing in the image of the clothing to be replaced, so as to obtain a target image of the model to be changed, wherein the image processing model is obtained by the above-mentioned image processing model training method.
[0021] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:
[0022] Memory and processor;
[0023] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above-mentioned image processing model training method and image processing method are implemented.
[0024] According to the sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the above-mentioned image processing model training method and image processing method are implemented.
[0025] According to the seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which implements the steps of the above-mentioned image processing model training method and image processing method when executed by a processor.
[0026] An image processing model training method provided by an embodiment of the present specification performs feature processing on a target object image and a simulated object image that has not been masked through a reference network of an image processing model to obtain target object reference features and simulated object reference features, wherein the target object reference features contain rich clothing information, and the simulated object reference features contain rich model information; determines a posture image of a simulated object in the simulated object image, and obtains a fused noise feature by fusing the posture features of the posture image with random noise, wherein the fused noise feature contains the posture information of the simulated object; when the target object reference features and the simulated object reference features are input into a backbone network of the image processing model, in the backbone network, denoising is performed on the fused noise features through the target object reference features and the simulated object reference features, and the posture information, clothing information and model information of the simulated object in the simulated object image are adaptively learned, so that the features of the required dressing area can be updated according to the target object reference features to obtain a predicted simulated object image, thereby avoiding the mask process that requires manual distinction of the image processing area, and coping with more complex image processing scenarios on the basis of achieving good background retention capabilities. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 is a scene schematic diagram of an image processing method provided by an embodiment of this specification;
[0028] Figure 2 is a flowchart of an image processing model training method provided by an embodiment of this specification;
[0029] Figure 3 is a flow chart of an image processing method provided by an embodiment of this specification;
[0030] Figure 4 is a processing flow chart of an image processing method provided by an embodiment of this specification;
[0031] Figure 5 is a model structure diagram of an image processing model provided by an embodiment of this specification;
[0032] Figure 6 It is a structural schematic diagram of an image processing model training device provided by an embodiment of this specification;
[0033] Figure 7 is a structural schematic diagram of an image processing device provided by an embodiment of this specification;
[0034] Figure 8 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION
[0035] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.
[0036] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0037] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0038] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0039] First, the terms involved in one or more embodiments of this specification are explained.
[0040] Virtual dressing: Virtual dressing is a virtual clothing technology generated by images. Users can preview the effects of different styles of clothing on their bodies through virtual dressing, so as to better choose the clothing that suits them.
[0041] inpainting: image completion.
[0042] Diffusion models: A class of image generation models.
[0043] Stable Diffusion: A vincemental graph model.
[0044] VAE: Variational Auto-Encoder, a variational autoencoder, an image encoding and decoding model.
[0045] Reference Unet: Reference network, a model for controllable generation of Stable Diffusion.
[0046] DINO: An image encoder that extracts semantic features of images and captures high-level abstract representations of images.
[0047] ResNet: Residual Network, a deep neural network structure, is mainly used to solve the gradient vanishing and gradient exploding problems that occur during the training of deep neural networks, as well as the problem of performance saturation or even degradation when the network depth increases.
[0048] Virtual dressing technology aims to provide users with visualized clothing wearing effects through digital images to improve the shopping experience. In the existing technology, through the inpainting paradigm, the accurate mask information of the dressing area provided is used to distinguish the foreground (mask area, usually the model's original clothing area) and the background (area outside the mask, usually the face, hands, background, etc. that need to remain unchanged) so as to complete the image in the mask area and realize dressing. However, the mask information needs to be manually specified or generated with the help of a segmentation model, which will incur more usage costs and increase uncertainty.
[0049] In practical applications, once deviation occurs in the mask area, the generation effect will be seriously affected; therefore, the embodiment of the present application specification provides a more advanced virtual fitting technology, which can introduce both the background and foreground into the image generation process through a reference network, automatically understand the area where the model needs to change clothes, avoid the link of providing mask information, and thus avoid the problem of poor quality of the generated image due to inaccurate mask information provided, thereby improving the generation effect of the generated image.
[0050] In this specification, an image processing model training method is provided. This specification also involves an image processing model training device, an image processing method, an image processing device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.
[0051] See also Figure 1 , Figure 1 A schematic diagram of a scene of an image processing method provided according to an embodiment of the present specification is shown.
[0052] Specifically, the image processing method is implemented by using the terminal device 102 and the server 104 , and the terminal device 102 is used to send the image of the model to be changed and the image of the clothing to be replaced to the server 104 .
[0053] An image processing model is trained in the server 104, and the image processing model includes a posture feature extraction network, a reference network, a clothing feature extraction network, and a backbone network; so that when the server 104 receives the image of the model to be changed and the image of the clothing to be replaced sent by the terminal device 102, the posture of the model in the image of the model to be changed is extracted to obtain a model posture image, the image of the model to be changed and the image of the clothing to be replaced are input into the reference network to obtain model reference features and clothing reference features, the model reference features and clothing reference features are injected into the backbone network, and the posture features obtained by extracting features from the model posture image through the posture feature extraction network and comparing them with random noise are characterized. After feature fusion, the fused noise features obtained by feature fusion are input into the backbone network. In the self-attention mechanism layer of the backbone network, the model information and clothing information are introduced by referring to the learned model reference features and clothing reference features to guide the image generation process. In the cross-attention mechanism layer of the backbone network, the target clothing features obtained by extracting clothing features from the clothing feature extraction network for the clothing image to be replaced can be introduced. The target clothing features include high-level semantic features of the clothing image to be replaced, so that the image generated by the image processing model can better retain high-level semantic information such as clothing style and material, generate a target changing model image with better quality, and return the target changing model image to the end-side device 102.
[0054] The end-side device 102 may include a browser, an APP (Application), or a web application such as an H5 (Hyper Text Markup Language 5, version 5 of Hypertext Markup Language) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The end-side device may be based on a software development kit (SDK) of the corresponding service provided by the server, such as based on the real-time communication (RTC, Real Time Communication) SDK development and acquisition. The end-side device may be deployed in an electronic device and needs to rely on the device to run or some APPs in the device to run. The electronic device may have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications may also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0055] Server 104 can be understood as a server that provides various services, including physical servers and cloud servers, such as a server that provides communication services to multiple clients, a server for background training that supports the model used on the client, and a server that processes the data sent by the client. It should be noted that server 104 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. Server 104 can also be a server for a distributed system, or a server combined with a blockchain. Server 104 can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.
[0056] It is worth noting that the image processing method provided in the embodiments of this specification can be executed by the server 104. In other embodiments of this specification, the image processing model can be deployed in the terminal device 102, so that the terminal device 102 can also have similar functions as the server 104, thereby executing the image processing method provided in the embodiments of this specification; in other embodiments, the image processing method provided in the embodiments of this specification can also be executed jointly by the terminal device 102 and the server 104.
[0057] An image processing method provided by an embodiment of the present specification, when the image of the clothing to be replaced and the image of the model to be dressed are both input into the reference network, uses the reference network to achieve accurate control of the clothing and background, and does not need to perform mask processing on the image of the model to be dressed to indicate the dressing area, that is, in the model reasoning process, the obvious distinction between the foreground and the background is eliminated, and there is no need to provide mask information. By obtaining high-level semantic features of the image of the clothing to be replaced, the image processing model's understanding of high-level semantic information is improved, thereby improving the quality of image generation.
[0058] See also Figure 2 , Figure 2 A flowchart of an image processing model training method provided by an embodiment of the present specification is shown, which specifically includes the following steps.
[0059] Step 202: Determine a target object image, a simulated object image, a posture image of a simulated object in the simulated object image, a target simulated object image, and random noise, wherein a reference object of the simulated object in the simulated object image is different from the target object in the target object image, and the target simulated object image is a simulated object image in which the reference object is replaced by the target object.
[0060] Among them, the target object image can be understood as a target clothing image, and the target object in the target object image can be understood as a target clothing in the target clothing image. The target clothing in the target clothing image is used to replace the clothing currently worn by the model. In the virtual fitting scene, the target clothing image is a tiled image of the clothing that the user wants to try on. The target clothing in the target clothing image includes but is not limited to clothing, hats, necklaces and other clothing items.
[0061] The simulated object image can be understood as a reference model image, which is a photo or image of a model used to display clothing. The simulated object in the simulated object image is the model in the reference model image. The model in the reference model image includes but is not limited to entities or images such as humans and cartoon characters that can wear and display clothing. The reference object of the simulated object in the simulated object image can be understood as the reference clothing worn by the model in the reference model image; the reference model image is used to provide a model carrier for replacing the target clothing image, that is, the clothing in the target clothing image can replace the clothing of the model in the reference model image to show the visual effect of the model in the reference model image wearing the clothing in the target clothing image; in a virtual fitting scene, the reference model image can be a full-body portrait uploaded by the user.
[0062] The posture image of the simulated object in the simulated object image is the posture image of the model in the reference model image, which can be understood as an image showing the movement posture of the model in the reference model image. The position, direction and scale information of each part of the model in the reference model image can be determined through the posture image.
[0063] The target simulation object image can be understood as the target model image, and the target model image can be understood as the model image obtained by replacing the reference clothing worn by the model in the reference model image with the target clothing in the target clothing image; the target model image is the target graph of the image processing model, that is, the goal of model training for the image processing model is that when the target clothing image, the reference model image, the posture image of the model in the reference model image, and random noise are input into the image processing model, the image processing model can output the target model image. The target model image is the same model as the model in the reference model image, and the posture of the model is also the same; in actual applications, the target model image and the reference model image are different except for the replaced clothing area, and the other background areas are consistent.
[0064] Random noise can be understood as Gaussian noise. Gaussian noise is a kind of random noise that obeys Gaussian distribution. The image is generated by denoising the random noise through the image processing model. When the random noise is used as the input for image generation, the random noise can be regarded as a potential feature vector and can be directly fused with other features.
[0065] Specifically, by determining the target clothing image, the reference model image, the model's posture image in the reference model image and random noise, training data for model training of the image processing model is obtained; by determining the target model image, label data for model training of the image processing model is obtained; the training data is input into the image processing model to obtain prediction results; and the image processing model is trained through the prediction results and label data.
[0066] In one or more embodiments of the present specification, since the target model image and the reference model image are images of the same model in the same posture wearing different clothes, and such data is difficult to collect, the real model image can be input into a conventional initial image processing model to generate a simulated model image, thereby determining the target model image and the reference model image based on the real model image and the simulated model image. The specific implementation is as follows:
[0067] The step of determining the simulated object image comprises:
[0068] Determine an image data set, wherein the image data set includes the target simulated object image and a replacement object image corresponding to the target simulated object image;
[0069] The target simulated object image and the replacement object image are input into an initial image processing model to obtain the simulated object image, wherein the initial image processing model is trained by sample target object images, sample simulated object images, mask simulated object images corresponding to the sample simulated object images, and label simulated object images, and the simulated object image is generated by replacing the target object of the simulated object in the target simulated object image with the replacement object in the replacement object image.
[0070] The target simulation object image can be understood as a real model image. When the real model image is not generated by a machine learning model, it can ensure that the image has a high clothing texture and retains more clothing details. The replacement object image is a replacement clothing image, which can be understood as a tiled image of clothing; the initial image processing model can be understood as an image processing model that implements clothing change based on the inpainting paradigm. When the initial image processing model is trained, the masked image is used for model training.
[0071] Specifically, model images and clothing images can be obtained from different data sources to construct an image dataset, which can contain a large number of grouped pictures of tops, bottoms, full-body clothing and corresponding models (that is, there is a corresponding target clothing image for a target model image); further, these pictures can be preprocessed, and the preprocessing includes but is not limited to model posture extraction, image cropping, image scaling, color adjustment and other processing to ensure image quality and consistency.
[0072] When the initial image processing model uses masked images for model training, the target simulated object image and the replacement object image are input into the initial image processing model. In fact, the target simulated object image must first be masked to indicate the dressing area that needs to be changed through the mask area, so as to complete the replacement object image to the dressing area of the target simulated object image to obtain a reference model image, which is a simulated model image obtained using the initial image processing model.
[0073] It should be noted that, when the reference model image is obtained by using the initial image processing model, the reference model image, the target clothing image corresponding to the target model image, and the posture image of the model in the reference model image (the posture image of the model in the reference model image is consistent with the posture image of the model in the target model image) are used as training data for training the image processing model, and the corresponding target model image is used as label data for the training image processing model. This is because the target model image is a real model image, the image has a higher clothing texture and retains more clothing details. When the target model image is used as label data, the quality of the image generated by the image processing model can be improved.
[0074] In practical applications, a data set covering a variety of styles and designs of clothing can be established through the data in the image data set and the initial image processing model. The data in the data set can be preprocessed and the image processing model can be trained using the data in the data set to enhance the generalization ability of the image processing model.
[0075] The image processing model training method provided in the embodiments of this specification utilizes an initial image processing model to process a target simulated object image in an image data set to obtain a reference model image, thereby determining training data for training the image processing model based on the target simulated object image in the image data set and the reference model image. When the target simulated object image in the image data set is a real model image, the quality of images generated by the image processing model can be improved.
[0076] In one or more embodiments of the present specification, when a reference model image is obtained, a posture estimation model may be used to extract the posture of the model in the reference model image, thereby obtaining a posture image of the model in the reference model image. The specific implementation is as follows:
[0077] The step of determining a posture image of a simulated object in the simulated object image comprises:
[0078] The simulated object image is input into a posture estimation model, and the posture estimation model is used to extract the posture of the simulated object in the simulated object image to obtain a posture image of the simulated object in the simulated object image.
[0079] The posture estimation model is used to identify and extract the action posture of the object in the input image, thereby generating a posture image that does not contain the appearance features of the object but contains the action posture of the object.
[0080] For example, when the simulated object in the simulated object image is a human, a posture estimation model, such as ViTPose (Vision Transformer Pose, a human posture estimation method), is used to perform human key point detection on the person in the simulated object image, mark the positions of the person's head, shoulders, hands, knees, feet, etc., and thus obtain a corresponding posture image.
[0081] In practical applications, the posture estimation model can realize the recognition and extraction of the action posture of the simulated object in the input image through different types of posture detectors. For example, the posture detector can be a two-dimensional skeleton key point detector, which is used to detect the skeleton key points and obtain the posture image of the model by connecting the skeleton key points; or, the posture detector can also be a three-dimensional action detector, which can fit the three-dimensional posture of the simulated object in the input image, and then obtain the posture image of the simulated object by rendering.
[0082] It should be noted that the purpose of obtaining the posture image in the embodiments of this specification is to obtain the posture information of the simulated object (the posture information describes the movement posture of the simulated object in space, that is, how the simulated object locates its body parts, joints, and overall shape and direction), and obtaining the posture image is a specific way to obtain the posture information of the simulated object. With the development of technology, it is not limited to using posture images to obtain the posture information of the simulated object, and other technical means can be used to obtain the posture information of the simulated object.
[0083] In addition, the above embodiment mentions that the posture image of the model in the reference model image is consistent with the posture image of the model in the target model image. Therefore, the target model image can be input into the posture estimation model, and the posture estimation model is used to extract the posture of the model in the target model image to obtain the posture image of the model in the target model image. When the target model image is processed to obtain the corresponding posture image, the acquisition of the posture image can be achieved when the data in the image data set is preprocessed.
[0084] The image processing model training method provided in the embodiments of this specification extracts the posture of the simulated object in the simulated object image through the posture estimation model, obtains the posture information of the simulated object in the simulated object image, thereby realizing fast and accurate generation of the posture image of the simulated object in the simulated object image.
[0085] Step 204: Using the reference network of the image processing model, perform feature processing on the target object image and the simulated object image respectively to obtain target object reference features and simulated object reference features, and input the target object reference features and the simulated object reference features into the backbone network of the image processing model.
[0086] Among them, the reference network can be understood as Reference U-Net, a model for controllable generation of Stable Diffusion. The backbone network has the same network structure and initialization parameters as the reference network; the backbone network can be understood as a denoising generation network for image generation.
[0087] The target object reference features and the simulated object reference features are reference features processed by the self-attention mechanism layer of the reference network and added with the source features, that is, the target object reference features are reference features processed by the self-attention mechanism layer of the reference network and added with the source features of the target object; the simulated object reference features are reference features processed by the self-attention mechanism layer of the reference network and added with the source features of the simulated object; in actual applications, when the target object is the target clothing and the simulated object is the model, the target object reference features and the simulated object reference features can be understood as clothing reference features and model reference features, respectively.
[0088] Specifically, the target clothing image and the reference model image are used as conditional inputs of the reference network, and the reference network is used to obtain clothing reference features corresponding to the target clothing image and model reference features of the reference model image; when the backbone network and the reference network have the same network structure and initialization parameters, it can be ensured that the features extracted by the reference network at different levels can be aligned with the backbone network. Therefore, the reference network can inject clothing reference features and model reference features into the backbone network, thereby introducing clothing information contained in the clothing reference features and model information contained in the model reference features into the backbone network.
[0089] In practical applications, both the reference network and the backbone network inherit the weights of the original SD (Stable Diffusion) UNet, so they can better maintain the basic ability of Stable Diffusion to denoise and generate images, and the generated image quality is also more natural.
[0090] In one or more embodiments of the present specification, the image processing model includes an image encoding and decoding network, and the reference network includes a reference self-attention mechanism layer; when the target clothing image and the reference model image are determined, image features are extracted for the target clothing image and the reference model image, so that the extracted image features are input into the reference network for processing. The specific implementation is as follows:
[0091] Before the reference network of the image processing model is used to perform feature processing on the target object image and the simulated object image to obtain the target object enhanced feature and the simulated object enhanced feature, the method further includes:
[0092] Using the image encoding and decoding network, respectively extracting image features of the target object image and the simulated object image to obtain target object image features and simulated object image features;
[0093] The reference network of the image processing model is used to perform feature processing on the target object image and the simulated object image respectively to obtain enhanced features of the target object and enhanced features of the simulated object, including:
[0094] The reference self-attention mechanism layer is used to perform self-attention mechanism processing on the target object image features and the simulated object image features respectively to obtain the target object enhanced features and the simulated object enhanced features.
[0095] The image encoding and decoding network can be understood as a VAE, which is used to encode an image to obtain image features, or decode an image feature to obtain an image; the target object image feature includes representative target object information extracted from the target object image, and the simulated object image feature includes representative simulated object information extracted from the simulated object image. Specifically, when the target object image is a target clothing image, the target object image feature is a clothing image feature corresponding to the target clothing image, including clothing information; when the simulated object image is a reference model image, the simulated object image feature is a model image feature, including model information.
[0096] The target object enhanced features can be understood as a set of enhanced target object feature representations obtained by processing the target object image features with a self-attention mechanism, and these target object feature representations are obtained by considering the interactions and dependencies between the target object image features. Similarly, the simulated object enhanced features can be understood as a set of enhanced simulated object feature representations obtained by processing the simulated object image features with a self-attention mechanism, and these simulated object feature representations are obtained by considering the interactions and dependencies between the simulated object image features.
[0097] Specifically, the purpose of adding a reference network on the basis of the backbone network is to introduce visual control conditions in the image generation process. The visual features of the target clothing image and the reference model image are extracted through the reference network, and then the visual features are input into the backbone network.
[0098] In practical applications, the reference network processes the image features of the target clothing image and the reference model image, that is, outside the reference network, an image encoding and decoding network is used to extract image features of the target clothing image and the reference model image, respectively, to obtain clothing image features containing clothing information and model image features containing model information, and then the clothing image features and the model image features are input into the reference network, and the self-attention mechanism layer in the reference network is used to perform self-attention mechanism processing on the clothing image features and the model image features, respectively, to obtain clothing enhancement features (that is, the target object enhancement features in the above embodiment) and model enhancement features (that is, the simulated object enhancement features in the above embodiment) that take into account the interaction and dependency between features.
[0099] The image processing model training method provided in the embodiments of this specification can capture rich contextual information in features through the processing of the self-attention mechanism, and the self-attention mechanism can dynamically adjust the attention weight according to the input features, automatically identify and focus on the important parts of the features, and ignore unimportant information. This adaptability improves the generalization ability and robustness of the image processing model.
[0100] In one or more embodiments of the present specification, when both the clothing reference features and the model reference features are input into the backbone network of the image processing model, in order to enable the backbone network to correctly distinguish the clothing reference features from the model reference features, source features indicating the source of the features are added to the clothing enhancement features and the model enhancement features. The specific implementation is as follows:
[0101] The reference network of the image processing model is used to perform feature processing on the target object image and the simulated object image respectively to obtain reference features of the target object and reference features of the simulated object, including:
[0102] Using a reference network of an image processing model, feature processing is performed on the target object image and the simulated object image respectively to obtain target object enhanced features and simulated object enhanced features;
[0103] Adding a target object source feature to the target object enhancement feature to obtain the target object reference feature;
[0104] The simulated object source feature is added to the simulated object enhancement feature to obtain the simulated object reference feature.
[0105] Specifically, when the clothing enhancement features and the model enhancement features are obtained through the reference network, the clothing enhancement features can be linearly transformed using three different weight matrices to obtain a key-value pair sequence corresponding to the clothing enhancement features, and the key-value pair sequence includes a Query, Key, and Value sequence; similarly, the model enhancement features can be linearly transformed using three different weight matrices to obtain a key-value pair sequence corresponding to the model enhancement features, and the key-value pair sequence includes a Query, Key, and Value sequence.
[0106] In the self-attention mechanism, Query can be regarded as a vector to be queried, which represents the information of the current position (or time step) and is used to obtain information from other positions; Key is also a vector, which is used to match with Query to determine which positions’ information is important to the current position; Value contains the actual information content in the feature.
[0107] In practical applications, the Key and Value sequences (key-value sequences) corresponding to the obtained clothing enhancement features and the Key and Value sequences corresponding to the model enhancement features will be injected from the reference self-attention mechanism layer of the reference network into the backbone self-attention mechanism layer of the backbone network. In order to distinguish the features of the target clothing image and the reference model image, that is, in order to distinguish the Key and Value sequences (key-value sequences) corresponding to the clothing enhancement features and the Key and Value sequences corresponding to the model enhancement features, source features are added to the Key sequences corresponding to the clothing enhancement features and the Key sequences corresponding to the model enhancement features, respectively. The source features are represented by learnable tokens to indicate the source of the features.
[0108] For example, the Key sequence corresponding to the clothing enhancement feature and the Key sequence corresponding to the model enhancement feature are two vectors, which can be recorded as k_cloth and k_person respectively. The clothing source feature (i.e., the target object source feature) and the model source feature (i.e., the simulation object source feature) are vectors of the same shape. The clothing source feature can be recorded as t_cloth to indicate that the content of the feature is the target clothing image, and the model source feature can be recorded as t_person to indicate that the content of the feature is the reference model image. By adding the two together, k_cloth+t_cloth and k_person+t_cloth respectively, we can get the Key sequence of clothing reference features containing the source features and the Key sequence of model reference features.
[0109] Therefore, the key-value sequence containing clothing source features and clothing reference features, as well as the key-value sequence containing model source features and clothing reference features are input into the backbone network of the image processing model; in this case, the backbone network can correctly distinguish clothing reference features from model reference features based on source features, avoiding mistaking model reference features for clothing reference features, thereby referring to and learning clothing information in model reference features, and failing to achieve the effect of model changing clothes.
[0110] The image processing model training method provided in the embodiments of this specification obtains clothing reference features by adding clothing source features to clothing enhancement features, and obtains model reference features by adding model source features to model enhancement features. When the clothing reference features and model reference features are input into the backbone network, the backbone network can learn accurate information from the correct features during the image generation process, thereby ensuring the generation quality of the image.
[0111] Step 206: Using the backbone network of the image processing model, feature processing is performed on the target object reference features, the simulated object reference features, and the fused noise features to obtain a predicted simulated object image, wherein the fused noise features are obtained by fusing the posture features of the posture image with the random noise.
[0112] Specifically, the posture features of the posture image are extracted, and the posture features are fused with random noise to obtain fused noise features, so that the fused noise features contain the posture information of the simulated object. In the backbone network, the fused noise features are denoised by learning the clothing reference features and model reference features transmitted from the reference network to obtain the predicted simulated object image.
[0113] In one or more embodiments of the present specification, the image processing model further includes a posture feature extraction network; the posture feature extraction network is used to extract posture features of the posture image, so as to guide the posture of the model in the image generation process through the posture features. The specific implementation is as follows:
[0114] Before the backbone network of the image processing model is used to perform feature processing on the target object reference feature, the simulated object reference feature and the fused noise feature to obtain the predicted simulated object image, the method further includes:
[0115] Using the posture feature extraction network, extracting posture features from the posture image to obtain the posture features;
[0116] The posture feature is fused with the random noise to obtain the fused noise feature.
[0117] Among them, since the backbone network has the same network structure as the reference network and also inherits the network design of Stable Diffusion, the core idea of Stable Diffusion is that since each image satisfies a certain regular distribution, the distribution information contained in the text is used as a guide to gradually denoise a pure noise image and generate an image that matches the text information; applying this idea, in the embodiment of this specification, the posture features of the posture image are fused with random noise and input into the backbone network, and denoising is performed using the backbone network, and clothing reference features and model reference features are introduced in the denoising process to guide the image generation process.
[0118] The pose feature extraction network can be understood as a pose guider, a lightweight component used to integrate the pose control signal (pose feature) into the denoising process. The pose guide uses a convolutional layer to extract the pose features of the pose image and align it with random noise of the same spatial resolution.
[0119] Specifically, the posture features corresponding to the posture image are extracted through the posture feature extraction network, and the posture features are added to the random noise (mentioned in the above embodiment, the random noise can be regarded as a potential feature vector), so that the processed posture image is added to the random noise before being input into the backbone network, thereby assisting the image generation process.
[0120] In practical applications, when the posture feature is added to the random noise feature, a fused noise feature is obtained, which contains the posture information of the simulated object. In the backbone network, the fused noise feature is used as input, and the target simulated object image is generated through iterative denoising. In the process of iterative denoising, the clothing reference features and model reference features of the reference network are introduced as control conditions to guide the generation of the target simulated object image.
[0121] The image processing model training method provided in the embodiments of the present specification obtains a fused noise feature by adding the posture feature and the random noise. The fused noise feature contains the posture information of the model. When the input of the backbone network is the fused noise feature, the backbone network iteratively denoises the fused noise feature to ensure the accuracy of the posture of the simulated object in the generated predicted simulated object image.
[0122] In one or more embodiments of the present specification, the backbone network includes a backbone self-attention mechanism layer and an image prediction layer; the above mentioned fusion noise features are used as the input of the backbone network, and the clothing reference features and model reference features of the reference network are integrated into the backbone network, and the clothing reference features, the model reference features and the fusion noise features are feature processed in the backbone network to obtain a predicted simulated object image. The specific implementation is as follows:
[0123] The method of utilizing the backbone network of the image processing model to perform feature processing on the target object reference feature, the simulated object reference feature and the fused noise feature to obtain a predicted simulated object image includes:
[0124] Using the backbone self-attention mechanism layer, obtaining a noise feature key-value pair sequence of the fused noise feature, wherein the noise feature key-value pair sequence includes a noise query sequence and a noise key-value sequence of the fused noise feature;
[0125] Determine a target object feature key-value pair sequence of the target object reference feature, and a simulation object feature key-value pair sequence of the simulation object reference feature, wherein the target object feature key-value pair sequence includes the target object key-value sequence of the target object reference feature, and the simulation object feature key-value pair sequence includes the simulation object key-value sequence of the simulation object reference feature;
[0126] Splicing the noise key value sequence with the target object key value sequence and the simulated object key value sequence to obtain a spliced key value sequence;
[0127] Obtaining predicted image features according to the noise query sequence and the concatenated key value sequence;
[0128] The image prediction layer is used to decode the predicted image features to obtain the predicted simulation object image.
[0129] In practical applications, the clothing enhancement features and the model enhancement features are input into the backbone self-attention mechanism layer. In the backbone self-attention mechanism layer, the fused noise features are spliced with the clothing enhancement features and the model enhancement features, and the self-attention mechanism is used to extract important information from the spliced features. Specifically, the weights of the spliced features containing posture information, clothing information, and model information are enhanced through the self-attention mechanism, and the weights of the features related to random noise in the spliced features are weakened, thereby achieving the effect of denoising the spliced features.
[0130] The specific processing process of the self-attention mechanism is that in the backbone self-attention mechanism layer of the backbone network, three different weight matrices can also be used to perform linear transformation on the fused noise features to obtain the noise feature key-value pair sequence corresponding to the fused noise features. The noise feature key-value pair sequence includes a noise query sequence (Query sequence) and a noise key-value sequence (Key and Value sequence).
[0131] Continuing with the above example, the above embodiment mentions that the Key and Value sequence (clothing key-value pair sequence) corresponding to the clothing reference feature and the Key and Value sequence (model key-value sequence) corresponding to the model reference feature are injected from the reference self-attention mechanism layer of the reference network into the backbone self-attention mechanism layer of the backbone network. On this basis, the noise key-value sequence is spliced with the clothing key-value sequence and the model key-value sequence to obtain a spliced key-value sequence. The noise query sequence that integrates the noise feature is used to calculate the similarity with the spliced key-value sequence, thereby updating the feature and obtaining the predicted image feature.
[0132] It should be noted that the similarity between the noise query sequence and the Key sequence in each concatenated key-value sequence (usually through dot product or cosine similarity, etc.) represents the correlation or importance between the current position and other positions. The similarity can be used to determine the weight of the corresponding position feature. For example, in the process of predicting the generation of simulated object images, the upper limb area of the simulated object needs to generate clothes. Then, the similarity calculated between the Query sequence of the fused noise feature corresponding to the upper limb area and the Key of the corresponding position in the clothing reference feature will be greater, thereby using more of the Value of the corresponding area of the target clothing image to update the features of the area that needs to generate clothes.
[0133] Specifically, when the predicted image features are obtained, in order to generate the predicted model image, the predicted image features need to be decoded to obtain a high-dimensional predicted simulation object image.
[0134] The image processing model training method provided in the embodiment of the present specification obtains predicted image features that introduce model information and clothing information by processing the key-value pair sequence corresponding to the clothing reference features, the model reference features, and the fused noise features in the backbone self-attention mechanism layer of the backbone network, thereby ensuring the generation quality of the generated predicted simulation object image.
[0135] In one or more embodiments of the present specification, attention weights are obtained by calculating the similarity between the noise query sequence and the concatenated key sequence, and then the features are weighted using the attention weights to obtain predicted image features. The specific implementation is as follows:
[0136] The step of obtaining predicted image features according to the noise query sequence and the concatenated key value sequence includes:
[0137] Calculate the similarity between the noise query sequence and the concatenated key sequence, and determine the attention weight corresponding to each key in the concatenated key sequence according to the obtained sequence similarity result;
[0138] The concatenated key value sequence is weighted according to the attention weight corresponding to each key value to obtain the predicted image feature.
[0139] Specifically, for each query (Query) element in the noise query sequence, calculate its similarity with each Key element in the concatenated key-value sequence to obtain the sequence similarity result, normalize the sequence similarity result, and obtain the attention weight of each position. These attention weights represent the importance of other position information at the current position. Use the normalized attention weights to perform weighted summation on the Value sequence in the concatenated key-value sequence to fuse the information and obtain the output of the current position. This output can be regarded as a feature representation of the predicted simulated object image, that is, the predicted image feature.
[0140] In practical applications, by performing self-attention processing on features in the backbone self-attention mechanism layer, the image processing model can adaptively adjust the weights of the clothing reference features and the model reference features in each region. The weights are determined by calculating the similarity. In the process of denoising and generating images, the weights of the features corresponding to the clothing key sequence and the model key sequence in the image generation area can be enhanced, while the weights of the features corresponding to the noise key sequence in the splicing key sequence can be weakened.
[0141] The image processing model training method provided in the embodiments of the present specification can adaptively adjust the weights of features by calculating the similarity between a noise query sequence and a spliced key value sequence, thereby enhancing the weights of clothing reference features and model reference features, thereby enhancing useful information such as clothing information and model information and suppressing noise information, thereby generating a higher quality predicted simulated object image through iterative denoising.
[0142] In one or more embodiments of the present specification, the image processing model also includes a target object feature extraction network; in order to enable the predicted simulation object image to better retain the high-level semantic information such as clothing style and material in the target clothing image, and improve the details of the target clothing in the predicted simulation object image, clothing features are extracted from the target clothing image, and the obtained target clothing features are injected into the backbone cross attention mechanism layer of the backbone network. The specific implementation method is as follows:
[0143] Before the backbone network of the image processing model is used to perform feature processing on the target object reference feature, the simulated object reference feature and the fused noise feature to obtain the predicted simulated object image, the method further includes:
[0144] Utilizing the target object feature extraction network, target object features are extracted from the target object image to obtain target object features, wherein the target object features include semantic information of the target object image.
[0145] Among them, the target object feature extraction network can be understood as a clothing feature extraction network, which is a network used to extract high-level features of target clothing in a target clothing image. For example, the target object feature extraction network is implemented using DINO. The extracted high-level features usually contain rich semantic information, so the high-level features can also be called semantic features. The semantic information of the target clothing includes but is not limited to high-level semantic information such as clothing style and material.
[0146] Specifically, by inputting the target clothing image into DINO, DINO is used to extract the high-level features of the target clothing in the target clothing image and use them as the input of the backbone cross-attention mechanism layer in the backbone network, thereby improving the high-level semantic understanding of the control condition of the target clothing in the backbone network.
[0147] The image processing model training method provided in the embodiments of this specification extracts clothing features from the target clothing image, injects the obtained target clothing features containing rich semantic information into the backbone cross-attention mechanism layer of the backbone network, and improves the detail quality of the clothing in the predicted model image.
[0148] In one or more embodiments of the present specification, the image processing model further includes an image prediction layer, and the backbone network includes a backbone self-attention mechanism layer and a backbone cross-attention mechanism layer; the target clothing features are injected into the backbone cross-attention mechanism layer of the backbone network, and the features output by the backbone self-attention mechanism layer and the target clothing features are processed by cross-attention mechanism in the backbone cross-attention mechanism layer to obtain predicted image features. The specific implementation is as follows:
[0149] The method of utilizing the backbone network of the image processing model to perform feature processing on the target object reference feature, the simulated object reference feature and the fused noise feature to obtain a predicted simulated object image includes:
[0150] Using the backbone self-attention mechanism layer, obtaining a noise feature key-value pair sequence of the fused noise feature, wherein the noise feature key-value pair sequence includes a noise query sequence and a noise key-value sequence of the fused noise feature;
[0151] Determine a target object feature key-value pair sequence of the target object reference feature, and a simulation object feature key-value pair sequence of the simulation object reference feature, wherein the target object feature key-value pair sequence includes the target object key-value sequence of the target object reference feature, and the simulation object feature key-value pair sequence includes the simulation object key-value sequence of the simulation object reference feature;
[0152] Splicing the noise key sequence with the target object key sequence and the simulated object key sequence to obtain a spliced key sequence, and obtaining an image enhancement feature according to the noise query sequence and the spliced key sequence;
[0153] Utilizing the backbone cross attention mechanism layer, calculating the similarity between the image enhancement feature and the target object feature, and determining the attention weight corresponding to the target object feature according to the obtained feature similarity result;
[0154] Performing weighted processing on the target object feature according to the attention weight corresponding to the target object feature to obtain the predicted image feature;
[0155] The image prediction layer is used to decode the predicted image features to obtain the predicted simulation object image.
[0156] For the specific implementation of obtaining the image enhancement feature, please refer to the above embodiment and will not be repeated here. The image enhancement feature in this embodiment is the feature output by the backbone self-attention mechanism layer. The similarity between the image enhancement feature and the target clothing feature (i.e., the target object feature) is calculated by an implementation method similar to the above to obtain the feature similarity result, and the attention weight corresponding to the image enhancement feature is determined according to the feature similarity result. The image enhancement feature is weighted according to the attention weight corresponding to the image enhancement feature to obtain the predicted image feature.
[0157] In practical applications, when the cross-attention mechanism is used to process image enhancement features and target clothing features, it is necessary to linearly transform the image enhancement features and the target clothing features to determine the Query sequence corresponding to the image enhancement features and the Key and Value sequences corresponding to the target clothing features, respectively, and calculate the similarity between each Query element in the Query sequence corresponding to the image enhancement features and all Key elements in the Key sequence corresponding to the target clothing features. According to the similarity calculation results, a normalization function is used to assign attention weights to each element in the target clothing features. These attention weights represent the degree of attention paid by the elements in the image enhancement features to the elements in the current target clothing features. The attention weights are used to weighted sum the Value sequences corresponding to the target clothing features to obtain the weighted feature representation, so as to integrate the information related to the image enhancement features in the target clothing features, and fuse or splice the feature representation obtained by the weighted sum with the image enhancement features to obtain the predicted image features.
[0158] The predicted image features are decoded using an image prediction layer. Specifically, the image prediction layer is an image decoder, which is used to decode the image features and restore them to a high-dimensional image. By decoding the predicted image features, a predicted simulation object image is obtained. The predicted simulation object image is a model image in which the reference clothing in the reference model image is replaced with the target clothing in the target clothing image.
[0159] The image processing model training method provided in the embodiments of this specification, when the target clothing features contain rich semantic information of the target clothing, processes the image enhancement features and the target clothing features using a cross-attention mechanism, so that the predicted image features output by the trunk cross-attention mechanism layer also contain rich semantic information of the target clothing, and uses the predicted image features to generate a predicted simulation object image, which can enable the predicted simulation object image to better retain high-level semantic information such as clothing style and material, thereby improving the image quality of the predicted simulation object image.
[0160] Step 208: Train the image processing model according to the predicted simulated object image and the target simulated object image to obtain a trained image processing model.
[0161] Specifically, by predicting the loss function of the predicted simulated object image and the target simulated object image, the model parameters of the image processing model are adjusted to obtain a trained image processing model. The predicted simulated object image output by the trained image processing model can be closer to restoring the target simulated object image.
[0162] In one or more embodiments of the present specification, the image processing model can be trained in two stages in actual applications. When the image processing model is trained in the first stage, the network parameters of part of the network in the image processing model can be adjusted. When the image processing model is trained in the second stage, all network parameters in the image processing model are adjusted. The specific implementation is as follows:
[0163] The reference network includes a reference self-attention mechanism layer and a reference cross-attention mechanism layer, and the backbone network includes a backbone self-attention mechanism layer and a backbone cross-attention mechanism layer;
[0164] The step of training the image processing model according to the predicted simulated object image and the target simulated object image to obtain the trained image processing model comprises:
[0165] According to the predicted simulated object image and the target simulated object image, adjusting the network parameters of the reference self-attention mechanism layer, the reference cross-attention mechanism layer, the backbone self-attention mechanism layer, and the backbone cross-attention mechanism layer until a preset training end condition is met, thereby obtaining an adjusted image processing model;
[0166] According to the predicted simulated object image obtained by the adjusted image processing model and the target simulated object image, the model parameters of the adjusted image processing model are adjusted to obtain a trained image processing model.
[0167] The preset training end condition may be to end the training when the training reaches a predetermined number of iterations (which can be set according to actual conditions), or to end the training when multiple consecutive verification performance indicators (such as accuracy, cross entropy loss, etc.) do not improve significantly when using the verification set to evaluate the model performance; there is no limitation here.
[0168] Specifically, according to the predicted simulated object image and the target simulated object image, the network parameters of the reference self-attention mechanism layer, the reference cross-attention mechanism layer, the trunk self-attention mechanism layer and the trunk cross-attention mechanism layer containing the self-attention mechanism and the cross-attention mechanism are adjusted to retain the Stable Diffusion capability of the basic model and obtain the adjusted image processing model. In practical applications, at this stage, on the basis of adjusting the network parameters of the above-mentioned networks, the network parameters of the posture feature extraction network can be adjusted to obtain the adjusted image processing model.
[0169] The adjusted image processing model is used to process the target clothing image, the reference model image, the model's posture image in the reference model image, and random noise to obtain a predicted simulation object image, thereby obtaining the predicted simulation object image and the target simulation object image through the adjusted image processing model, adjusting all network parameters of the adjusted image processing model, and obtaining a trained image processing model, so that the trained image processing model can be better applicable to the dressing task.
[0170] The image processing model training method provided in the embodiments of this specification can greatly improve the training efficiency of the image processing model and the generation effect of the generated image through a two-stage training method.
[0171] The image processing model training method provided in the embodiments of the present specification does not require masking of the reference model image. By inputting the reference model image and the target clothing image into the reference network, the reference network is used to obtain the model reference features containing the model information and the clothing reference features containing the clothing information, thereby achieving accurate control of the clothing and the background (usually the areas that need to remain unchanged, such as the face, hands, and background). When the model reference features and the clothing reference features are integrated into the backbone network, and the target clothing features containing the high-level semantic information of the clothing are integrated into the backbone network, in the backbone network, the self-attention mechanism and the cross-attention mechanism are used to adaptively refer to the model information, clothing information, and the high-level semantic information of clothing to generate an image with better quality.
[0172] See also Figure 3 , Figure 3 A flowchart of an image processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.
[0173] Step 302: Determine the model image to be changed and the clothing image to be replaced.
[0174] Among them, the model image to be changed can be understood as a model image in which the clothes of the model in the image will be replaced; the clothing image to be replaced is a tiled image of the clothing to be replaced, which can show the clothing details of the clothing to be replaced, and the clothing to be replaced is used to replace the clothing of the model in the model image to be changed.
[0175] Specifically, a model image to be changed and an image of clothing to be replaced are determined, and the effect of changing the model's clothing is achieved by replacing the clothing to be replaced in the image of clothing to be replaced with the clothing of the model in the image of clothing to be changed.
[0176] In actual applications, if the image of the clothing to be replaced is a tiled image of a top, then the image of the model to be dressed up at least includes the upper body area of the model; or if the image of the clothing to be replaced is a tiled image of a bottom, then the image of the model to be dressed up at least includes the lower body area of the model, that is, the area displayed by the model in the image of the model to be dressed up at least includes the area corresponding to the clothing to be replaced.
[0177] Step 304: Input the image of the model to be changed and the image of the clothing to be replaced into an image processing model, and use the image processing model to replace the clothing in the image of the model to be changed with the clothing in the image of the clothing to be replaced, so as to obtain a target image of the model to be changed, wherein the image processing model is obtained by the above-mentioned image processing model training method.
[0178] Specifically, the posture of the model in the model image to be changed is extracted to obtain the model posture image, the model image to be changed and the clothing image to be replaced are input into the reference network of the image processing model to obtain the model reference feature and the clothing reference feature, and the model reference feature and the clothing reference feature are injected into the backbone network; the model posture image is feature extracted by the posture feature extraction network to obtain the posture feature, the posture feature is fused with random noise and then input into the backbone network, in the self-attention mechanism layer of the backbone network, the model information and the clothing information are introduced by referring to the model reference feature and the clothing reference feature to guide the image generation process, the image enhancement feature output by the self-attention mechanism layer of the backbone network is used as the input of the cross-attention mechanism layer of the backbone network, in the cross-attention mechanism layer of the backbone network, the target clothing feature obtained by extracting the clothing feature of the clothing image to be replaced by the clothing feature extraction network is introduced, the target clothing feature contains the high-level semantic features of the clothing image to be replaced, the target predicted image feature output by the cross-attention mechanism layer of the backbone network is obtained, and the target changed model image is obtained by using the target predicted image feature; the target changed model image is obtained by replacing the clothing in the model image to be changed with the clothing in the clothing image to be replaced.
[0179] For example, the image of the model to be changed is an image of user A wearing a skirt, and the image of the clothing to be replaced is a tiled image of a pair of jeans. Through the processing of the image processing model, the target changing model image of user A wearing the jeans can be obtained. In the target changing model image, except that the skirt of user A is replaced with jeans, the rest of the background is consistent with the image of the model to be changed.
[0180] In one or more embodiments of the present specification, the user can send the image of the model to be changed and the image of the clothing to be replaced to the server through the client through the interactive operation with the user interaction interface of the client, and after obtaining the target image of the model to be changed, the target image of the model to be changed can be returned to the client to display the target image of the model to the user through the user interaction interface of the client. The specific implementation is as follows:
[0181] The step of determining the model image to be changed and the clothing image to be replaced includes:
[0182] The image of the model to be changed and the image of the clothing to be replaced sent by the client are received, wherein the image of the model to be changed and the image of the clothing to be replaced are sent in the case of interactive operation with a user interaction interface of the client.
[0183] After obtaining the target dress-changing model image, the method further includes:
[0184] The target dress-changing model image is returned to the client, so as to be displayed on a user interaction interface of the client.
[0185] Specifically, some preset images of models to be changed and images of clothing to be replaced can be displayed on the user interaction interface of the client, so that the user can directly click to select from the user interaction interface; or the user can also click the upload button on the user interaction interface of the client to upload the images of models to be changed and / or images of clothing to be replaced in the user's local area.
[0186] The interactive operation of the user on the user interaction interface of the client triggers the client to send the image of the model to be changed and the image of the clothing to be replaced to the server, so that the server receives the image of the model to be changed and the image of the clothing to be replaced sent by the client.
[0187] When the target dressed-up model image is returned to the client, a visual dressing effect is provided to the user through the user interaction interface of the client. In the e-commerce platform scenario, this method can improve the user's shopping experience.
[0188] The image processing method provided in the embodiment of the present specification triggers the client to send the image of the model to be changed and the image of the clothing to be replaced through the user's interactive operation on the user interaction interface of the client, which can be more convenient for users to use and improve the user experience by displaying the target model image to be changed on the user interaction interface of the client.
[0189] An image processing method provided by an embodiment of the present specification, when the image of the clothing to be replaced and the image of the model to be dressed are both input into the reference network, uses the reference network to achieve accurate control of the clothing and background, and does not need to perform mask processing on the image of the model to be dressed to indicate the dressing area, that is, in the model reasoning process, the obvious distinction between the foreground and the background is eliminated, and there is no need to provide mask information. By obtaining high-level semantic features of the image of the clothing to be replaced, the image processing model's understanding of high-level semantic information is improved, thereby improving the quality of image generation.
[0190] See also Figure 4 , Figure 4 A processing flow chart of an image processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.
[0191] Step 402: Build a data set.
[0192] Using image data obtained from different data sources, a large-scale virtual fitting dataset (i.e., the image dataset in the above embodiment) is constructed, which includes a large number of group images of tops, bottoms, full-body outfits and corresponding models; these images are preprocessed, including but not limited to posture extraction, cropping, scaling and color adjustment, to ensure their quality and consistency.
[0193] Step 404: Prepare simulation data.
[0194] Since the embodiments of this specification require a group of pictures of the same model in the same pose wearing different clothes as training data, and such data is difficult to collect, the conventional dressing model is trained using the existing inpainting dressing method, and the conventional dressing model is used to perform arbitrary dressing on the real model pictures in the virtual fitting data set (i.e., the target model images in the above embodiments), thereby obtaining the required simulated model pictures (i.e., the reference model images in the above embodiments).
[0195] Among them, the real model picture is the target image generated by the image processing model, the target clothing image corresponding to the model clothing in the real model picture and the simulated model image are the conditional inputs of the reference network in the image processing model, and the posture image corresponding to the simulated model image (i.e., the posture image in the above embodiment) is the conditional input of the backbone network, and the above constitute a set of training data.
[0196] Step 406: Train the model.
[0197] Determine the model structure of the image processing model. For details, see Figure 5 , Figure 5 A model structure diagram of an image processing model provided by an embodiment of the present specification is shown.
[0198] A reference network is added on the basis of the backbone network (the network structure of Stable Diffusion). The reference network is used to introduce visual control conditions. The reference network uses the same network architecture and initialization parameters as the backbone network, so it can ensure that the features extracted by the reference network at different levels can be aligned with the backbone network.
[0199] The target clothing image and the simulated model image are respectively extracted with VAE for image features, and visual features are respectively extracted through the reference network. For example, through the self-attention layer of the reference network, the clothing enhancement features corresponding to the target clothing image and containing clothing information and the reference enhancement features corresponding to the simulated model image and containing model information are obtained. The Key and Value sequences corresponding to the clothing enhancement features and the reference enhancement features are spliced with the Key and Value sequences of the backbone network in the self-attention layer. In order to distinguish the features of the target clothing image and the simulated model image, learnable tokens (i.e., the clothing source features and the model source features in the above embodiment) are added to the corresponding Keys of the two to indicate the source of the features.
[0200] In addition, the posture features corresponding to the model posture graph (obtained by extracting the posture guide composed of convolutional layers, the posture features include the model posture information) and the initialized random noise are input into the backbone network, wherein the model posture information is injected into the backbone network to assist the generation process; in order to improve the backbone network's high-level semantic understanding of clothing conditions, the clothing feature extraction network is used to extract the high-level features of the target clothing image and inject them into the cross-attention layer of the backbone network; in order to ensure the consistency between the backbone network and the reference network, additional conditions also need to be introduced into the cross-attention layer of the reference network, and the additional conditions introduced in the basic model of Stable Diffusion are texts, so the cross-attention layer of the reference network requires text input. In the embodiment of this specification, an empty string is used as the conditional input in the cross-attention layer of the reference network, that is, the empty string is input into the text feature extraction layer, and no additional text conditions are introduced; in addition, similar to the backbone network, the high-level features of the target clothing image can be used as additional conditional inputs of the cross-attention layer, but experiments have shown that the effect is roughly consistent with that of using an empty string as the conditional input, so in order to improve the training and reasoning efficiency of the model, an empty string is used as the conditional input.
[0201] In actual applications, in the reference network, the target clothing image passes through the ResNet layer, the self-attention layer, and the cross-attention layer in sequence, where an empty string is used as the conditional input in the cross-attention layer without introducing additional text conditions; for the simulated model image, the reference network is used for similar processing.
[0202] In the backbone network, the noisy image features (random noise in the above embodiment) and the model posture graph are used as inputs, and the target graph is generated by iterative denoising. Specifically, the posture features of the model posture graph are extracted using a posture guide, and added to the noisy image features, and input into the ResNet layer, self-attention layer, and cross-attention layer of the backbone network, so as to predict and generate the image. In the self-attention layer of the backbone network, the Query, Key, and Value sequence corresponding to the noisy image is obtained, and the Key and Value sequences obtained in the corresponding self-attention layer of the reference network for the simulated model image and the target clothing image are spliced with the Key and Value sequences of the noisy image, and the Query of the noisy image and the spliced Key and Value are used to calculate the similarity and update the features, thereby introducing the model information of the simulated model image and the clothing information of the target clothing image; in the cross-attention layer of the backbone network, the high-level features of the target clothing image extracted by the clothing feature extraction network are injected, so that the image generated by the image processing model can better retain the high-level semantic information such as clothing style and material.
[0203] The ResNet layer, self-attention layer, and cross-attention layer are taken as a group of network units. The backbone network and the reference network include multiple groups of network units composed of ResNet layers, self-attention layers, and cross-attention layers. The features output by the self-attention layer in each network unit of the reference network are integrated into the corresponding self-attention layer of the backbone network, and the features output by the last group of network units in the backbone network are used as the predicted image features.
[0204] Since the inputs of the reference network and the backbone network are both features, the output of the backbone network is the predicted image features. By decoding the predicted image features, the predicted model image is obtained, and the image processing model is trained using the predicted model image and the real model image.
[0205] The training is divided into two stages. The first stage trains the posture guide and the layers containing the self-attention mechanism and the cross-attention mechanism in the backbone network and the reference network. In the second stage, all parameters in the backbone network and the reference network are added to the training. This training method can greatly retain the image generation ability of the pre-trained model, thereby ensuring the generation effect of the image generation model.
[0206] It should be noted that steps 402 to 406 describe a method for training an image processing model in the image processing method. After the image processing model is obtained through training in steps 402 to 406, the image processing method of steps 408 to 410 is implemented using the trained image processing model.
[0207] Step 408: Determine the image.
[0208] Determine the image of the model to be changed and the image of the clothing to be replaced; the specific implementation is shown in the above embodiment and will not be repeated here.
[0209] Step 410: Generate an image.
[0210] The image of the model to be changed and the image of the clothing to be replaced are input into the image processing model, and the clothing in the image of the model to be changed is replaced with the clothing in the image of the clothing to be replaced by using the image processing model to generate a target image of the model to be changed.
[0211] In practical applications, the image processing model can be integrated into the e-commerce platform, so that when users shop online on the e-commerce platform, they can upload images of their own image and images of the clothes they want to buy before shopping, and check the wearing effect of the clothes in advance before purchase, realizing virtual fitting, thereby improving the user's shopping experience; or merchants can use the image processing model to generate model images, and then use the generated model images to decorate the store, saving the merchants the model costs and shooting costs of model fitting, etc., thereby improving the merchants' user experience.
[0212] Furthermore, the method can also be applied to a visual technology capability application platform, thereby achieving a virtual dressing effect on the visual technology capability application platform, so that users are not limited to realizing virtual trying on clothes in shopping scenarios, and can also be applied to other scenarios. For example, many ID photos have specific dressing requirements, for example, dark-colored collared clothing is required. In this case, users can upload their existing ID photos that do not meet the dressing requirements and dark-colored collared clothing pictures on the visual technology capability application platform, thereby obtaining ID photos that meet the above dressing requirements.
[0213] The image processing method provided in the embodiments of this specification is not only technologically advanced in the field of academic research, but also successfully transformed into practical application cases. The image processing method can be integrated into various platforms to achieve feasibility in real scenarios and has great potential in improving the online shopping experience.
[0214] The image processing method provided in the embodiments of this specification effectively improves the generalization ability of the model by constructing a large-scale virtual fitting data set, ensuring good results on clothing of different styles and designs; and realizes accurate control of clothing and background at the same time. This method avoids the display distinction process of foreground and background during the reasoning process, thereby eliminating the need to provide mask information.
[0215] Corresponding to the above method embodiment, this specification also provides an image processing model training device embodiment, Figure 6 FIG. 1 is a schematic diagram showing the structure of an image processing model training device provided by an embodiment of the present specification. Figure 6 As shown, the device comprises:
[0216] An image determination module 602 is configured to determine a target object image, a simulated object image, a posture image of a simulated object in the simulated object image, a target simulated object image, and random noise, wherein a reference object of the simulated object in the simulated object image is different from a target object in the target object image, and the target simulated object image is a simulated object image in which the reference object is replaced by the target object;
[0217] The feature processing module 604 is configured to perform feature processing on the target object image and the simulated object image respectively using the reference network of the image processing model to obtain target object reference features and simulated object reference features, and input the target object reference features and the simulated object reference features into the backbone network of the image processing model;
[0218] The image acquisition module 606 is configured to perform feature processing on the target object reference feature, the simulated object reference feature and the fused noise feature using the backbone network of the image processing model to obtain a predicted simulated object image, wherein the fused noise feature is obtained by fusing the posture feature of the posture image with the random noise;
[0219] The model training module 608 is configured to train the image processing model according to the predicted simulated object image and the target simulated object image to obtain a trained image processing model.
[0220] Optionally, the feature processing module 604 is further configured to:
[0221] Using a reference network of an image processing model, feature processing is performed on the target object image and the simulated object image respectively to obtain target object enhanced features and simulated object enhanced features;
[0222] Adding a target object source feature to the target object enhancement feature to obtain the target object reference feature;
[0223] The simulated object source feature is added to the simulated object enhancement feature to obtain the simulated object reference feature.
[0224] The device also includes: a feature extraction module configured to use the image encoding and decoding network to extract image features of the target object image and the simulated object image respectively, so as to obtain target object image features and simulated object image features.
[0225] Optionally, the feature processing module 604 is further configured to:
[0226] The reference self-attention mechanism layer is used to perform self-attention mechanism processing on the target object image features and the simulated object image features respectively to obtain the target object enhanced features and the simulated object enhanced features.
[0227] The device also includes: a posture extraction module, which is configured to use the posture feature extraction network to extract posture features from the posture image to obtain the posture features; and fuse the posture features with the random noise to obtain the fused noise features.
[0228] Optionally, the image acquisition module 606 is further configured to:
[0229] Using the backbone self-attention mechanism layer, obtaining a noise feature key-value pair sequence of the fused noise feature, wherein the noise feature key-value pair sequence includes a noise query sequence and a noise key-value sequence of the fused noise feature;
[0230] Determine a target object feature key-value pair sequence of the target object reference feature, and a simulation object feature key-value pair sequence of the simulation object reference feature, wherein the target object feature key-value pair sequence includes the target object key-value sequence of the target object reference feature, and the simulation object feature key-value pair sequence includes the simulation object key-value sequence of the simulation object reference feature;
[0231] Splicing the noise key value sequence with the target object key value sequence and the simulated object key value sequence to obtain a spliced key value sequence;
[0232] Obtaining predicted image features according to the noise query sequence and the concatenated key value sequence;
[0233] The image prediction layer is used to decode the predicted image features to obtain the predicted simulation object image.
[0234] Optionally, the image acquisition module 606 is further configured to:
[0235] Calculate the similarity between the noise query sequence and the concatenated key sequence, and determine the attention weight corresponding to each key in the concatenated key sequence according to the obtained sequence similarity result;
[0236] The concatenated key value sequence is weighted according to the attention weight corresponding to each key value to obtain the predicted image feature.
[0237] The device also includes: a clothing extraction module, configured to use the target object feature extraction network to extract target object features from the target object image to obtain target object features, wherein the target object features include semantic information of the target object image.
[0238] Optionally, the image acquisition module 606 is further configured to:
[0239] Using the backbone self-attention mechanism layer, obtaining a noise feature key-value pair sequence of the fused noise feature, wherein the noise feature key-value pair sequence includes a noise query sequence and a noise key-value sequence of the fused noise feature;
[0240] Determine a target object feature key-value pair sequence of the target object reference feature, and a simulation object feature key-value pair sequence of the simulation object reference feature, wherein the target object feature key-value pair sequence includes the target object key-value sequence of the target object reference feature, and the simulation object feature key-value pair sequence includes the simulation object key-value sequence of the simulation object reference feature;
[0241] Splicing the noise key sequence with the target object key sequence and the simulated object key sequence to obtain a spliced key sequence, and obtaining an image enhancement feature according to the noise query sequence and the spliced key sequence;
[0242] Utilizing the backbone cross attention mechanism layer, calculating the similarity between the image enhancement feature and the target object feature, and determining the attention weight corresponding to the target object feature according to the obtained feature similarity result;
[0243] Performing weighted processing on the target object feature according to the attention weight corresponding to the target object feature to obtain the predicted image feature;
[0244] The image prediction layer is used to decode the predicted image features to obtain the predicted simulation object image.
[0245] Optionally, the image determination module 602 is further configured to:
[0246] Determine an image data set, wherein the image data set includes the target simulated object image and a replacement object image corresponding to the target simulated object image;
[0247] The target simulated object image and the replacement object image are input into an initial image processing model to obtain the simulated object image, wherein the initial image processing model is trained by sample target object images, sample simulated object images, mask simulated object images corresponding to the sample simulated object images, and label simulated object images, and the simulated object image is generated by replacing the target object of the simulated object in the target simulated object image with the replacement object in the replacement object image.
[0248] Optionally, the model training module 608 is further configured to:
[0249] According to the predicted simulated object image and the target simulated object image, adjusting the network parameters of the reference self-attention mechanism layer, the reference cross-attention mechanism layer, the backbone self-attention mechanism layer, and the backbone cross-attention mechanism layer until a preset training end condition is met, thereby obtaining an adjusted image processing model;
[0250] According to the predicted simulated object image obtained by the adjusted image processing model and the target simulated object image, the model parameters of the adjusted image processing model are adjusted to obtain a trained image processing model.
[0251] Optionally, the image determination module 602 is further configured to:
[0252] The simulated object image is input into a posture estimation model, and the posture estimation model is used to extract the posture of the simulated object in the simulated object image to obtain a posture image of the simulated object in the simulated object image.
[0253] An image processing model training device provided by an embodiment of the present specification performs feature processing on a target object image and a simulated object image that has not been masked through a reference network of an image processing model to obtain a target object reference feature and a simulated object reference feature, wherein the target object reference feature contains rich clothing information, and the simulated object reference feature contains rich model information; determines a posture image of a simulated object in the simulated object image, and obtains a fused noise feature by fusing the posture feature of the posture image with random noise, wherein the fused noise feature contains the posture information of the simulated object; when the target object reference feature and the simulated object reference feature are input into a backbone network of the image processing model, in the backbone network, the fused noise feature is denoised through the target object reference feature and the simulated object reference feature, and the posture information, clothing information and model information of the simulated object in the simulated object image are adaptively learned, so that the features of the required dressing area can be updated according to the target object reference feature to obtain a predicted simulated object image, thereby avoiding the mask process that requires manual distinction of the dressing area, and coping with more complex dressing scenes on the basis of achieving good background retention capabilities.
[0254] The above is a schematic scheme of an image processing model training device of this embodiment. It should be noted that the technical scheme of the image processing model training device and the technical scheme of the above-mentioned image processing model training method belong to the same concept, and the details not described in detail in the technical scheme of the image processing model training device can be found in the description of the technical scheme of the above-mentioned image processing model training method.
[0255] Corresponding to the above method embodiment, this specification also provides an image processing device embodiment, Figure 7 FIG. 2 shows a schematic diagram of the structure of an image processing device provided by an embodiment of the present specification. Figure 7 As shown, the device comprises:
[0256] An image determination module 702 is configured to determine an image of a model to be changed and an image of clothing to be replaced;
[0257] The image acquisition module 704 is configured to input the image of the model to be changed and the image of the clothing to be replaced into an image processing model, and use the image processing model to replace the clothing in the image of the model to be changed with the clothing in the image of the clothing to be replaced, so as to obtain a target image of the model to be changed, wherein the image processing model is obtained by the above-mentioned image processing model training method.
[0258] Optionally, the image determination module 702 is further configured to:
[0259] The image of the model to be changed and the image of the clothing to be replaced sent by the client are received, wherein the image of the model to be changed and the image of the clothing to be replaced are sent in the case of interactive operation with a user interaction interface of the client.
[0260] The device further includes: an image returning module configured to return the target dressing model image to the client, so as to display the target dressing model image on a user interaction interface of the client.
[0261] An image processing device provided by an embodiment of the present specification utilizes an image processing model, and when both the image of the clothing to be replaced and the image of the model to be replaced are input into a reference network, the reference network is used to achieve accurate control of the clothing and background, and no mask processing is required to indicate the clothing change area for the image of the model to be replaced, that is, in the model reasoning process, the obvious distinction between the foreground and the background is eliminated, and there is no need to provide mask information. By obtaining high-level semantic features of the image of the clothing to be replaced, the image processing model's understanding of high-level semantic information is improved, thereby improving the quality of image generation.
[0262] The above is a schematic scheme of an image processing device of this embodiment. It should be noted that the technical scheme of the image processing device and the technical scheme of the above-mentioned image processing method belong to the same concept, and the details not described in detail in the technical scheme of the image processing device can be referred to the description of the technical scheme of the above-mentioned image processing method.
[0263] Figure 8 The block diagram of a computing device 800 according to an embodiment of the present specification is shown. The components of the computing device 800 include but are not limited to a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.
[0264] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.
[0265] In one embodiment of the present specification, the above components of the computing device 800 and Figure 8 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 8 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0266] The computing device 800 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 800 may also be a mobile or stationary server.
[0267] Among them, the processor 820 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-mentioned image processing model training method and image processing method.
[0268] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the image processing model training method and the image processing method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the image processing model training method and the image processing method embodiment.
[0269] An embodiment of the present specification also provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned image processing model training method and image processing method when executed by a processor.
[0270] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the image processing model training method and the image processing method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the image processing model training method and the image processing method embodiment.
[0271] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned image processing model training method and image processing method.
[0272] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned image processing model training method and image processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned image processing model training method and image processing method.
[0273] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0274] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0275] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.
[0276] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0277] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.
Claims
1. A method for training an image processing model, comprising: Determine a target object image, a simulated object image, a posture image of a simulated object in the simulated object image, a target simulated object image, and random noise, wherein a reference object of the simulated object in the simulated object image is different from a target object in the target object image, and the target simulated object image is a simulated object image in which the reference object is replaced by the target object; Using a reference network of an image processing model, feature processing is performed on the target object image and the simulated object image respectively to obtain target object reference features and simulated object reference features, and the target object reference features and the simulated object reference features are input into a backbone network of the image processing model; Using the backbone network of the image processing model, feature processing is performed on the target object reference feature, the simulated object reference feature and the fused noise feature to obtain a predicted simulated object image, wherein the fused noise feature is obtained by fusing the posture feature of the posture image with the random noise; The image processing model is trained according to the predicted simulated object image and the target simulated object image to obtain a trained image processing model.
2. The image processing model training method according to claim 1, wherein the reference network of the image processing model is used to perform feature processing on the target object image and the simulated object image to obtain target object reference features and simulated object reference features, respectively, including: Using a reference network of an image processing model, feature processing is performed on the target object image and the simulated object image respectively to obtain target object enhanced features and simulated object enhanced features; Adding a target object source feature to the target object enhancement feature to obtain the target object reference feature; The simulated object source feature is added to the simulated object enhancement feature to obtain the simulated object reference feature.
3. The image processing model training method according to claim 2, wherein the image processing model further comprises an image encoding and decoding network, and the reference network comprises a reference self-attention mechanism layer; Before the reference network of the image processing model is used to perform feature processing on the target object image and the simulated object image to obtain the target object enhanced feature and the simulated object enhanced feature, the method further includes: Using the image encoding and decoding network, respectively extracting image features of the target object image and the simulated object image to obtain target object image features and simulated object image features; The reference network of the image processing model is used to perform feature processing on the target object image and the simulated object image respectively to obtain enhanced features of the target object and enhanced features of the simulated object, including: The reference self-attention mechanism layer is used to perform self-attention mechanism processing on the target object image features and the simulated object image features respectively to obtain the target object enhanced features and the simulated object enhanced features.
4. The image processing model training method according to claim 1, wherein the image processing model further comprises a posture feature extraction network; Before the backbone network of the image processing model is used to perform feature processing on the target object reference feature, the simulated object reference feature and the fused noise feature to obtain the predicted simulated object image, the method further includes: Using the posture feature extraction network, extracting posture features from the posture image to obtain the posture features; The posture feature is fused with the random noise to obtain the fused noise feature.
5. According to the image processing model training method of claim 4, the backbone network includes a backbone self-attention mechanism layer and an image prediction layer; The method of utilizing the backbone network of the image processing model to perform feature processing on the target object reference feature, the simulated object reference feature and the fused noise feature to obtain a predicted simulated object image includes: Using the backbone self-attention mechanism layer, obtaining a noise feature key-value pair sequence of the fused noise feature, wherein the noise feature key-value pair sequence includes a noise query sequence and a noise key-value sequence of the fused noise feature; Determine a target object feature key-value pair sequence of the target object reference feature, and a simulation object feature key-value pair sequence of the simulation object reference feature, wherein the target object feature key-value pair sequence includes the target object key-value sequence of the target object reference feature, and the simulation object feature key-value pair sequence includes the simulation object key-value sequence of the simulation object reference feature; Splicing the noise key value sequence with the target object key value sequence and the simulated object key value sequence to obtain a spliced key value sequence; Obtaining predicted image features according to the noise query sequence and the concatenated key value sequence; The image prediction layer is used to decode the predicted image features to obtain the predicted simulation object image.
6. The image processing model training method according to claim 5, wherein obtaining predicted image features according to the noise query sequence and the concatenated key value sequence comprises: Calculate the similarity between the noise query sequence and the concatenated key sequence, and determine the attention weight corresponding to each key in the concatenated key sequence according to the obtained sequence similarity result; The concatenated key value sequence is weighted according to the attention weight corresponding to each key value to obtain the predicted image feature.
7. The image processing model training method according to claim 1, wherein the image processing model further comprises a target object feature extraction network; Before the backbone network of the image processing model is used to perform feature processing on the target object reference feature, the simulated object reference feature and the fused noise feature to obtain the predicted simulated object image, the method further includes: Utilizing the target object feature extraction network, target object features are extracted from the target object image to obtain target object features, wherein the target object features include semantic information of the target object image.
8. The image processing model training method according to claim 7, wherein the image processing model further comprises an image prediction layer, and the backbone network comprises a backbone self-attention mechanism layer and a backbone cross-attention mechanism layer; The method of utilizing the backbone network of the image processing model to perform feature processing on the target object reference feature, the simulated object reference feature and the fused noise feature to obtain a predicted simulated object image includes: Using the backbone self-attention mechanism layer, obtaining a noise feature key-value pair sequence of the fused noise feature, wherein the noise feature key-value pair sequence includes a noise query sequence and a noise key-value sequence of the fused noise feature; Determine a target object feature key-value pair sequence of the target object reference feature, and a simulation object feature key-value pair sequence of the simulation object reference feature, wherein the target object feature key-value pair sequence includes the target object key-value sequence of the target object reference feature, and the simulation object feature key-value pair sequence includes the simulation object key-value sequence of the simulation object reference feature; Splicing the noise key sequence with the target object key sequence and the simulated object key sequence to obtain a spliced key sequence, and obtaining an image enhancement feature according to the noise query sequence and the spliced key sequence; Utilizing the backbone cross attention mechanism layer, calculating the similarity between the image enhancement feature and the target object feature, and determining the attention weight corresponding to the target object feature according to the obtained feature similarity result; Performing weighted processing on the target object features according to the attention weights corresponding to the target object features to obtain predicted image features; The image prediction layer is used to decode the predicted image features to obtain the predicted simulation object image.
9. The image processing model training method according to claim 1, wherein determining the simulated object image comprises: Determine an image data set, wherein the image data set includes the target simulated object image and a replacement object image corresponding to the target simulated object image; The target simulated object image and the replacement object image are input into an initial image processing model to obtain the simulated object image, wherein the initial image processing model is trained by sample target object images, sample simulated object images, mask simulated object images corresponding to the sample simulated object images, and label simulated object images, and the simulated object image is generated by replacing the target object of the simulated object in the target simulated object image with the replacement object in the replacement object image.
10. The image processing model training method according to claim 1, wherein the reference network comprises a reference self-attention mechanism layer and a reference cross-attention mechanism layer, and the backbone network comprises a backbone self-attention mechanism layer and a backbone cross-attention mechanism layer; The step of training the image processing model according to the predicted simulated object image and the target simulated object image to obtain the trained image processing model comprises: According to the predicted simulated object image and the target simulated object image, adjusting the network parameters of the reference self-attention mechanism layer, the reference cross-attention mechanism layer, the backbone self-attention mechanism layer, and the backbone cross-attention mechanism layer until a preset training end condition is met, thereby obtaining an adjusted image processing model; According to the predicted simulated object image obtained by the adjusted image processing model and the target simulated object image, the model parameters of the adjusted image processing model are adjusted to obtain a trained image processing model.
11. The image processing model training method according to any one of claims 1 to 10, wherein determining the posture image of the simulated object in the simulated object image comprises: The simulated object image is input into a posture estimation model, and the posture estimation model is used to extract the posture of the simulated object in the simulated object image to obtain a posture image of the simulated object in the simulated object image.
12. An image processing method, comprising: Determine the image of the model to be changed and the image of the clothing to be replaced; The image of the model to be changed and the image of the clothing to be replaced are input into an image processing model, and the clothing in the image of the model to be changed is replaced with the clothing in the image of the clothing to be replaced using the image processing model to obtain a target image of the model to be changed, wherein the image processing model is obtained by any one of the image processing model training methods of claims 1-11.
13. The image processing method according to claim 12, wherein determining the model image to be changed and the clothing image to be replaced comprises: Receiving the image of the model to be changed and the image of the clothing to be replaced sent by the client, wherein the image of the model to be changed and the image of the clothing to be replaced are sent in the case of interactive operation with a user interaction interface of the client; After obtaining the target dress-changing model image, the method further includes: The target dress-changing model image is returned to the client, so as to be displayed on a user interaction interface of the client.
14. An image processing model training device, comprising: an image determination module, configured to determine a target object image, a simulated object image, a posture image of a simulated object in the simulated object image, a target simulated object image, and random noise, wherein a reference object of the simulated object in the simulated object image is different from a target object in the target object image, and the target simulated object image is a simulated object image in which the reference object is replaced by the target object; a feature processing module configured to perform feature processing on the target object image and the simulated object image respectively using a reference network of an image processing model, obtain target object reference features and simulated object reference features, and input the target object reference features and the simulated object reference features into a backbone network of the image processing model; An image acquisition module is configured to perform feature processing on the target object reference feature, the simulated object reference feature and the fused noise feature using the backbone network of the image processing model to obtain a predicted simulated object image, wherein the fused noise feature is obtained by fusing the posture feature of the posture image with the random noise; The model training module is configured to train the image processing model according to the predicted simulated object image and the target simulated object image to obtain a trained image processing model.
15. An image processing device, comprising: An image determination module is configured to determine an image of a model to be changed and an image of clothing to be replaced; The image acquisition module is configured to input the image of the model to be changed and the image of the clothing to be replaced into an image processing model, and use the image processing model to replace the clothing in the image of the model to be changed with the clothing in the image of the clothing to be replaced, so as to obtain a target image of the model to be changed, wherein the image processing model is obtained by any one of the image processing model training methods of claims 1-11.
16. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 13 are implemented.
17. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13.
18. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13.
