Virtual fitting video generation method, model training method and electronic equipment

Abstract

The invention discloses a virtual fitting video generation method, a model training method, electronic equipment and a computer readable storage medium. The video generation method comprises the following steps: acquiring a human body video and a costume picture corresponding to target costume; extracting target clothing features and human body features corresponding to the video frames; fusing the target clothing feature with each human body feature in a spatial dimension; performing feature adjustment on the target clothing feature based on each human body feature to obtain each human body matching clothing feature matched with the human body in each video frame; and fusing each space fusion feature and each human body matching clothing feature in a time dimension to obtain a time fusion feature, and obtaining each human body try-on video frame on which the target clothing is tried according to the time fusion feature. Through the scheme provided by the invention, the generated virtual try-on video has better inter-frame continuity and fluency, so that the authenticity of the final virtual try-on video is improved, and the virtual try-on effect is better.

Description

Technical Field

[0001] The present application relates to the field of virtual try-on technology, and in particular to a virtual try-on video generation method, a model training method, a device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In today's digital age, e-commerce platforms are rapidly becoming the preferred channel for consumers to shop. This trend has given rise to an urgent need for online shopping experience technology. For clothing products, virtual try-on technology has attracted much attention because it allows customers to preview clothing effects without actually wearing them. In order to improve the user's virtual try-on experience, clothing can be virtually tried on in a human body video to form a virtual try-on video, so that users can view their own try-on effects through dynamic videos, providing a better user experience.

[0003] In the related art, the image of the virtual clothing is usually fused with each frame of the human body video, so that each frame is set as a video frame of trying on the virtual clothing to obtain a virtual try-on video.

[0004] However, since the video requires a coherent transition between frames, the related technology approach usually has certain errors in the clothing fusion of each frame, and the fusion effect of each frame is relatively independent. For example, the lighting effects between the two frames are different, the clothing details between the two frames are different, etc., which easily makes the inter-frame continuity and smoothness of the fused virtual try-on video poor, resulting in poor authenticity of the final virtual try-on video and poor virtual try-on effect. Summary of the invention

[0005] The present application provides a virtual try-on video generation method, an image generation model training method, a device, an electronic device, and a computer-readable storage medium, which can make the generated virtual try-on video have better inter-frame continuity and smoothness, thereby improving the authenticity of the final virtual try-on video and achieving a better virtual try-on effect. The specific scheme is as follows:

[0006] In a first aspect, the present application provides a method for generating a virtual try-on video, the method comprising:

[0007] Obtaining a human body video to be virtually tried on and a clothing picture corresponding to the target clothing;

[0008] Extracting target clothing features corresponding to the target clothing from the clothing picture, and extracting human body features corresponding to each video frame from the human body video;

[0009] The target clothing features are respectively fused with the human body features in the spatial dimension to obtain spatial fusion features;

[0010] Based on the human body features, the target clothing features are adjusted to obtain human body matching clothing features that match the human body in each video frame;

[0011] Each of the spatial fusion features is fused with each of the human body matching clothing features in the time dimension to obtain a time fusion feature, and each human body fitting video frame wearing the target clothing is obtained based on the time fusion feature.

[0012] In a second aspect, the present application also provides a model training method, the method comprising:

[0013] Acquire training samples, wherein the training samples include a sample human body video, a sample clothing picture corresponding to a sample target clothing, and a sample fitting video of a person fitting the sample target clothing;

[0014] Extracting sample target clothing features corresponding to the sample target clothing from the sample clothing picture, and extracting sample human body features corresponding to each sample video frame from the sample human body video;

[0015] Based on each of the sample human body features, feature adjustment is performed on the sample target clothing features to obtain each sample human body matching clothing feature that matches the human body in each sample video frame;

[0016] Performing feature processing on the sample human body features corresponding to each frame through each feature processing layer of the image generation model to be trained, obtaining output processed human body features corresponding to each frame that are integrated with the sample target clothing features and the features of each frame are consistent in time series, and determining each output human body fitting video frame fitting the sample target clothing according to the output processed human body features;

[0017] Based on the consistency between the output human fitting video frame and each sample video frame of the sample fitting video, adjusting the parameters of the image generation model to be trained to obtain a trained image generation model;

[0018] The input of the i-th feature processing layer of the image generation model to be trained is obtained by the following steps:

[0019] The output of the i-1th feature processing layer corresponding to each sample frame is fused with the sample target clothing feature in the spatial dimension to obtain the i-1th spatial fusion feature of the sample corresponding to each sample frame;

[0020] The i-1th spatial fusion feature of the sample corresponding to each sample frame is fused with the clothing feature matching each sample body in the time dimension to obtain the i-1th time fusion feature of the sample, and the i-1th time fusion feature of the sample is determined as the input of the i-th feature processing layer, i traverses 1 to N, N is the number of feature processing layers included in the image generation model to be trained, and the output of the 0th feature processing layer is the sample body feature.

[0021] In a third aspect, the present application also provides a model training method, the method comprising:

[0022] Acquire training samples, wherein the training samples include a sample human body video, a sample clothing picture corresponding to a sample target clothing, and a sample fitting video of a person fitting the sample target clothing;

[0023] Extracting sample target clothing features corresponding to the sample target clothing from the sample clothing picture, and extracting sample human body features corresponding to each sample video frame from the sample human body video;

[0024] The sample human body features corresponding to each frame are processed by each feature processing layer of the image generation model to be trained to obtain the first output processed human body features integrated with the sample target clothing features, and the first output human body fitting video frames trying on the sample target clothing are determined according to the first output processed human body features; wherein the input of the i-th feature processing layer of the image generation model to be trained is obtained by the following steps: the output of the i-1th feature processing layer corresponding to each sample frame is integrated with the sample target clothing features in the spatial dimension to obtain the i-1th spatial fusion features of the samples corresponding to each sample frame, and the input of the i-th feature processing layer is determined according to the i-1th spatial fusion features, i traverses 1 to N, N is the number of feature processing layers included in the image generation model to be trained, and the output of the 0th feature processing layer is the sample human body features;

[0025] Based on the consistency between the first output human fitting video frame and each sample video frame of the sample fitting video, adjusting the parameters of the image generation model to be trained to obtain the image generation model after the first stage of training;

[0026] Based on each of the sample human body features, feature adjustment is performed on the sample target clothing features to obtain each sample human body matching clothing feature that matches the human body in each sample video frame;

[0027] The sample human body features corresponding to each frame are processed by each feature processing layer of the image generation model after training in the first stage, and the second output processed human body features corresponding to each frame are obtained, which are integrated with the sample target clothing features and the features of each frame are consistent in the time series, and the second output human body fitting video frames trying on the sample target clothing are determined according to the second output processed human body features; wherein the input of the i-th feature processing layer of the image generation model after training in the first stage is obtained by the following steps: the output of the i-1th feature processing layer corresponding to each sample frame is integrated with the sample target clothing features in the spatial dimension to obtain the sample i-1th spatial fusion feature corresponding to each sample frame; the sample i-1th spatial fusion feature corresponding to each sample frame is integrated with each sample human body matching clothing feature in the time dimension to obtain the sample i-1th time fusion feature, and the sample i-1th time fusion feature is determined as the input of the i-th feature processing layer, i traverses 1 to N, N is the number of feature processing layers included in the image generation model to be trained, and the 0th sample spatial fusion feature is the sample human body feature;

[0028] Based on the consistency between the second output human fitting video frame and each sample video frame of the sample fitting video, the parameters of the image generation model after training in the first stage are adjusted to obtain the trained image generation model.

[0029] In a fourth aspect, the present application also provides an electronic device, comprising: a processor, a memory, and computer program instructions stored in the memory and executable on the processor; when the processor executes the computer program instructions, it implements the method described in any one of the first to third aspects.

[0030] In a fifth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement any one of the methods in the first to third aspects.

[0031] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method as described in any one of the first to third aspects.

[0032] Compared with the prior art, this application has the following advantages:

[0033] The virtual try-on video generation method provided in the embodiment of the present application obtains a human body video to be virtually tried on and a clothing picture corresponding to a target clothing, extracts target clothing features corresponding to the target clothing from the clothing picture, and extracts human body features corresponding to each video frame from the above human body video, and fuses the target clothing features with each human body feature in the spatial dimension to obtain spatial fusion features corresponding to each video frame. The spatial fusion features can well represent the features after the human body features in each video frame are combined with the target clothing features, so as to represent the basic appearance display of the target clothing being tried on the human body in the video frame of the human body video, and then adjusts the target clothing features based on each human body feature to obtain each human body matching clothing feature that matches the human body in each video frame. The human body matching clothing feature can match the human body posture and human body structure in the corresponding video frame, so as to make the virtual try-on more effective. The result is more in line with the human body state. For example, when the human body is twisting its body posture, the human body matching clothing feature is the state feature of the target clothing when the human body is twisting its body posture. Then, each spatial fusion feature is fused with each human body matching clothing feature in the time dimension. Since each spatial fusion feature is the feature after each video frame of the human body video is fused with the target clothing feature, therefore, each spatial fusion feature is fused with each human body matching clothing feature in the time dimension, that is, the target clothing feature is indirectly fused with the human body feature of each video frame in the time dimension, which can better ensure that the target clothing feature is consistent with each frame of the human body video in the time dimension, so that the obtained time fusion feature corresponding to each frame can not only well reflect the effect of a single video frame trying on the target clothing, but also make each human body trying on the target clothing obtained according to the above time fusion feature can also maintain the consistency and consistency of each frame in the time dimension.

[0034] It can be seen that the solution provided by the present application can well generate the effect of trying on the target clothing in a single video frame by fusing the target clothing features with the human body features in the spatial dimension, and can achieve accurate retention of clothing details. By fusing each spatial fusion feature with each human body matching clothing feature in the temporal dimension, the coherence and consistency of each frame can be maintained in the temporal dimension, so that the generated virtual try-on video has better inter-frame continuity and smoothness, thereby improving the authenticity of the final virtual try-on video and achieving a better virtual try-on effect. The present application not only improves the generation quality of single-frame images, but also ensures the natural transition and true performance of clothing in the entire video sequence, thereby providing users with a more realistic and coherent virtual try-on experience.

[0035] The model training method provided in the present application obtains training samples, and the training samples include sample human body videos, sample clothing pictures corresponding to sample target clothing, and sample fitting videos of the above-mentioned sample target clothing being tried on, extracts sample target clothing features corresponding to the sample target clothing from the sample clothing pictures, and extracts sample human body features corresponding to each sample video frame from the sample human body video; performs feature adjustment on the sample target clothing features based on each sample human body feature, and obtains each sample human body matching clothing feature that matches the human body in each sample video frame; performs feature processing on the sample human body features corresponding to each frame through each feature processing layer of the image generation model to be trained, and obtains output processed human body features corresponding to each frame that are fused with the sample target clothing features and whose features are consistent in time series, and determines each output human body fitting video of trying on the sample target clothing based on the output processed human body features. frequency frames; wherein, the input of the i-th feature processing layer of the image generation model to be trained is obtained by the following steps: the output of the i-1th feature processing layer corresponding to each sample frame is fused with the sample target clothing feature in the spatial dimension to obtain the sample i-1th spatial fusion feature corresponding to each sample frame; the sample i-1th spatial fusion feature corresponding to each sample frame is fused with the sample body matching clothing feature in the time dimension to obtain the sample i-1th time fusion feature, and the sample i-1th time fusion feature is determined as the input of the i-th feature processing layer, i traverses 1 to N, N is the number of feature processing layers included in the image generation model to be trained, and the output of the 0th feature processing layer is the sample body feature; based on the consistency between the output human body fitting video frame and each sample video frame of the sample fitting video, the parameters of the image generation model to be trained are adjusted to obtain the trained image generation model. In the process of training the image generation model of the present application, the control model can obtain the output processed human body features corresponding to each frame that are integrated with the sample target clothing features and the features of each frame are consistent in the time series, so that the trained image generation model can not only well guarantee the effect of trying on the target clothing of the generated single video frame, but also enable the video frames generated by the image generation model to maintain the coherence and consistency of each frame in the time dimension, so that the virtual try-on video generated by the image generation model trained by the method provided by the present application has better inter-frame continuity and smoothness, thereby improving the authenticity of the final virtual try-on video and achieving a better virtual try-on effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the application scenario of the virtual try-on video generation solution provided by this application;

[0037] Figure 2 It is a flowchart of an example of a method for generating a virtual try-on video provided in an embodiment of the present application;

[0038] Figure 3It is a display diagram of human body features in the embodiments of the present application;

[0039] Figure 4 It is a schematic diagram of a model processing flow of a virtual try-on video generation method provided in an embodiment of the present application;

[0040] Figure 5 yes Figure 4 A schematic diagram of a process for performing data fusion in the process of providing a method for generating a virtual try-on video;

[0041] Figure 6 It is a flowchart of an example of a model training method provided in an embodiment of the present application;

[0042] Figure 7 It is a flowchart of another example of the model training method provided in the embodiment of the present application;

[0043] Figure 8 It is a structural block diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the technical solution of the present application, the present application is described clearly and completely below in conjunction with the drawings in the embodiments of the present application. However, the present application can be implemented in many other ways different from the following description. Therefore, based on the embodiments provided by the present application, all other embodiments obtained by those of ordinary skill in the art without creative work should fall within the scope of protection of the present application.

[0045] It should be noted that the terms "first", "source domain", "third", etc. in the claims, description and drawings of the present application are used to distinguish similar objects and are not used to describe a specific order or sequence. The data used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having" and their variants are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] In order to facilitate understanding of the various embodiments of the present application, the application background of the embodiments is described.

[0047] In today's digital age, e-commerce platforms are rapidly becoming the preferred channel for consumers to shop. This trend has given rise to an urgent need for online shopping experience technology. For clothing products, virtual try-on technology has attracted much attention because it allows customers to preview clothing effects without actually wearing them. In order to improve the user's virtual try-on experience, clothing can be virtually tried on in a human body video to form a virtual try-on video, so that users can view their own try-on effects through dynamic videos, providing a better user experience.

[0048] In the related art, the image of the virtual clothing is usually fused with each frame of the human body video, so that each frame is set as a video frame of trying on the virtual clothing to obtain a virtual try-on video.

[0049] However, since the video requires a coherent transition between frames, the related technology approach usually has certain errors in the clothing fusion of each frame, and the fusion effect of each frame is relatively independent. For example, the lighting effects between the two frames are different, the clothing details between the two frames are different, etc., which easily makes the inter-frame continuity and smoothness of the fused virtual try-on video poor, resulting in poor authenticity of the final virtual try-on video and poor virtual try-on effect.

[0050] In order to solve the above problems, the embodiments of the present application provide a virtual try-on video generation method, a model training method, a device, an electronic device and a computer-readable storage medium, which are intended to enable the generated virtual try-on video to have better inter-frame continuity and smoothness, thereby improving the authenticity of the final virtual try-on video and achieving a better virtual try-on effect.

[0051] The virtual try-on video generation method provided in this application can be used to generate a virtual try-on video, specifically, to form a coherent and accurate virtual try-on video for each frame of a human body trying on a target garment in a human body video, based on a human body video shot and uploaded by a user. The solution provided in this application can be used to try on garments for sale in an e-commerce platform, and can also be used for virtual try-ons in other scenarios, such as video production platforms, virtual game platforms, clothing design platforms, etc. This application does not specifically limit the specific application scenarios for generating virtual try-on videos.

[0052] In order to facilitate understanding of the method embodiments of the present application, the application scenarios are introduced. Figure 1 , Figure 1 Schematic diagram of the application scenario of the solution provided in the embodiment of the present application. For the convenience of description, the following describes the embodiment of the present application by taking the e-commerce platform as an example. The application scenario is a schematic illustration and is not intended to be a specific description of the application scenario. Figure 1As shown, in this application scenario, a server 102 and a client 101 are provided. In this embodiment, a connection is established between the client 101 and the server 102 via network communication, so as to perform data transmission.

[0053] The client 101 can be an electronic device with display and data processing functions, such as a mobile phone, a tablet computer (pad), a smart watch, a desktop computer, a smart TV, a VR device, a vehicle-mounted device, a wearable device, a laptop computer, etc. The client 101 is used to send the human body video uploaded by the user and the clothing picture corresponding to the target clothing selected by the user to the server 102, and obtain and play the human body fitting video generated by the server 102 from the server 102.

[0054] The server 102 has high computing power. The server 102 can be a server, which has high-speed processor (central processing unit, CPU) computing power, long-term reliable operation, powerful input / output (input / output, I / O) external data throughput capacity and better scalability. The server 102 can be a single server or a server cluster. The server 102 is used to receive human body videos and clothing pictures corresponding to target clothing from the client 101, and generate corresponding human body try-on videos based on the human body videos and clothing pictures. The server 102 can also provide other specific services for the client 101, such as user information access, website access, application access, etc., which are not specifically limited in this application.

[0055] The client 101 and the server 102 may communicate with each other using various communication systems, such as a wired communication system or a wireless communication system. The wireless communication system may be, for example, a global system for mobile communications (GSM) system, a code division multiple access (CDMA) system, a wideband code division multiple access (WCDMA) system, a general packet radio service (GPRS), a long term evolution (LTE) system, a LTE frequency division duplex (FDD) system, a LTE time division duplex (TDD) system, a universal mobile telecommunication system (UMTS), a worldwide interoperability for microwave access (WiMAX) communication system, a future fifth generation (5G) system or a new radio (NR), a satellite communication system, etc.

[0056] Embodiment 1

[0057] The first embodiment of the present application provides a method for generating a virtual try-on video, which is applied to an electronic device, which may be a server, a laptop, a tablet computer, a desktop computer, a mobile phone, a smart TV, a smart hand, a VR device, a client, and other electronic devices with data processing functions.

[0058] like Figure 2 As shown, the virtual try-on video generation method provided in the first embodiment of the present application includes the following steps S110 to S150.

[0059] Step S110: obtaining a human body video to be virtually tried on and a clothing picture corresponding to the target clothing.

[0060] The above-mentioned human body video can be a video containing the human body recorded and uploaded by a user who needs to perform virtual try-on, or it can be a human body video containing a model or other human body stored in an electronic device, or it can be other videos containing the human body, which is not specifically limited in this application.

[0061] The clothing picture corresponding to the target clothing can be a clothing picture selected by the user who needs to perform virtual try-on through a selection operation, or a clothing picture uploaded by the user to the electronic device, or a clothing picture pre-set by the electronic device, which is not specifically limited in this application.

[0062] Step S120: extracting target clothing features corresponding to the target clothing from the clothing picture corresponding to the target clothing, and extracting human body features corresponding to each video frame from the human body video.

[0063] The above-mentioned target clothing features may include clothing-intensive features such as color features, shape features, texture features, pattern features, outline features, size features, and decorative element features of the target clothing, but are not limited thereto. Color features may include the color distribution and proportion of the target clothing, shape features may include the outline, cutting style (such as straight lines, curves), etc. of the target clothing, texture features may include the texture of the fabric of the target clothing (such as smooth, rough), pattern features may include the pattern type (such as stripes, plaids, flowers) and arrangement of the target clothing, outline features may include the overall outline of the clothing, detailed outlines such as the neckline and cuffs, size features may include the size and proportion (such as skirt length, sleeve length), etc. of the clothing, and decorative element features may include features of additional decorations such as buttons, zippers, and embroidery.

[0064] In the present application embodiment, Figure 4 , Figure 5 As shown, in step S120, the garment image can be garment encoded by a pre-trained garment encoding (Garment Encoder) model to obtain the target garment features corresponding to the target garment. The garment encoding model is used to extract the garment features corresponding to the garments in the garment image from the garment image. The garment encoding model can be a neural network model, a diffusion model, etc., or other machine learning models. The garment encoding model can be obtained by training with a supervised learning algorithm, a semi-supervised learning algorithm, or an unsupervised learning algorithm. The specific training method will be introduced later. Those skilled in the art can also use the training method in the relevant technology for training, which is not specifically limited in this application. In this embodiment, the garment encoding model can quickly and accurately extract the garment features of each dimension from the garment image through the pre-trained garment encoding model, thereby improving the efficiency and accuracy of feature extraction, thereby improving the efficiency and try-on effect of virtual try-on.

[0065] Optionally, in an embodiment of the present application, other methods may be used to extract target clothing features corresponding to the target clothing from the clothing image. For example, clothing contours may be extracted by edge detection technology, clothing color features may be extracted by color histogram technology, features may be extracted from clothing images using some open source tools, etc., which are not specifically limited in the present application.

[0066] The human features corresponding to the video frames may include features such as body contour, human key points, human body shape, human body movement trajectory, human body posture, etc., and may also include other human features, which are not specifically limited in this application.

[0067] In a specific embodiment, step S120 can extract the human body features corresponding to each video frame through the following step S121.

[0068] Step S121: masking the clothing area in the video frame of the human body video to obtain a clothing-agnostic video frame, and determining the clothing-agnostic video frame as a human body feature corresponding to the video frame.

[0069] The hand area in the clothing-agnostic video frame is in an unmasked state. The clothing area refers to an area for trying on clothing, and the clothing area may include an arm area, a torso area, a leg area, a foot area, and other areas where clothing needs to be worn.

[0070] Since the clothing-agnostic video frames contain the hand area, the human body posture can be judged more accurately based on the hand area, making the video frames of the virtual try-on more accurate and realistic.

[0071] This embodiment uses clothing-agnostic video frames as human body features. Since the clothing area is masked in the clothing-agnostic video frames, the interference of existing clothing in the video frames can be eliminated to better display the target clothing. In addition, removing the original clothing can make the display of the target clothing clearer and avoid confusion. By using the agnostic video frames, the shape and dynamics of the target clothing can be applied more accurately.

[0072] Specifically, step S121 can identify the clothing region from the video frame of the human body by using a pre-trained clothing region recognition model, and then mask the clothing region.

[0073] In a specific embodiment, step S120 may also extract the human features corresponding to each video frame through the following step S122.

[0074] Step S122: extracting a human body analysis result of each video frame from the human body video, and determining the human body analysis result as a human body feature corresponding to the video frame.

[0075] The human body analysis result is used to distinguish and mark various parts of the human body in the video frame. Specifically, the human body analysis result of each frame can be extracted through human body analysis algorithms such as Graphinomy, or through other methods, such as DeepCut algorithm, Self-Correction for Human Parsing (Schp) algorithm, M2FP algorithm, etc. This application will no longer introduce the method of determining the human body analysis result in detail. The human body analysis result can mark the pixel position of each part in the image and the body part to which it belongs, providing a fine geometric foundation for applications such as human body understanding, posture analysis and action recognition. The human body analysis result can be a multi-channel mask image, in which each pixel is marked as belonging to a specific part of the body, such as the head, hand, leg, etc. The human body analysis result can provide precise positioning and boundary information about various parts of the human body. Figure 3 As shown, Figure 3 The video frame containing a human body shown in Figure 3 The human body analysis result corresponding to the first original image in Figure 3 As shown in the third image, from Figure 3 As can be seen from the third image in the figure, the human body analysis results can identify different areas of the human body through different channel colors. In the virtual fitting task, because the human body analysis results provide the corresponding parts and wearing information of the human body, we can better understand the posture of the human body and the area where the clothing is located, and then provide a mask to cover the clothing area, eliminate the interference of existing clothing in the video, and then try the target clothing on the human body.

[0076] In a specific embodiment, step S120 may also extract the human body features corresponding to each video frame through the following step S123.

[0077] Step S123: determining a mask corresponding to the clothing agnostic video frame, and determining the mask as a human body feature corresponding to the video frame.

[0078] The mask is used to distinguish the human body area and the masked clothing area in the clothing-agnostic video frame. The mask corresponding to the clothing-agnostic video frame can be a binary image, which is used to indicate which areas in the video frame are clothing and which areas are background or human body, such as Figure 3 As shown, Figure 3 The first image in the video frame containing a human body (i.e. Figure 3 The mask corresponding to the clothing agnostic video frame corresponding to the original image in Figure 3 As shown in the fourth figure in the figure, the mask corresponding to the clothing-agnostic video frame distinguishes the area used to wear clothing from other areas in the original image. Figure 3As can be seen from the fourth figure in the figure, the mask corresponding to the clothing-agnostic video frame can identify the area where the clothing needs to be tried on and other areas through white and black pixel values. The mask corresponding to the agnostic video frame can more accurately determine the area where the clothing needs to be replaced, ensuring that the clothing being tried on is accurately displayed in the specified clothing area. It can also maintain the natural connection between the clothing and the human body and avoid unnatural transitions. By accurately controlling the display position of the clothing, it can better simulate the actual wearing effect and improve the user experience.

[0079] Steps S121 to S123, namely, the human body features corresponding to the video frame include the clothing agnostic video frame, the human body analysis result and the mask.

[0080] In a specific embodiment, a clothing-agnostic video frame can be determined based on the above-mentioned human body analysis results. Since the human body analysis results distinguish and mark the areas corresponding to different parts of the human body in the video frame, the clothing area corresponding to the video frame can be conveniently determined based on the human body analysis results, thereby masking the clothing area in the video frame to obtain a clothing-agnostic video frame.

[0081] In a specific embodiment, the following steps A to C may also be used to obtain clothing agnostic video frames.

[0082] Step A: determining the human body key points, human body posture information and human body analysis results corresponding to each frame from the video frames of the human body video.

[0083] Among them, the human body key points corresponding to each frame refer to the key feature points of the human body in the video frame, usually including the head, shoulders, neck, elbows, wrists, hips, knees, ankles, finger joints and other position points. In the embodiment of the present application, the human body key points can be extracted from the video frame by the DWpose algorithm, wherein the DWpose algorithm can achieve efficient human body key point detection through the convolutional neural network (CNN), and can accurately identify key parts in various complex scenes. Other algorithms can also be used to identify human body key points, such as the OpenPose algorithm, PoseNet, etc., which are not specifically limited in this application. Figure 3 As shown, Figure 3 The human key points corresponding to the video frame containing the human body shown in the first figure are as follows Figure 3 As shown in the second figure in the figure. The determination method and related meaning of the human body analysis results can be referred to in the previous text, and will not be repeated here.

[0084] Human body posture information is used to represent the posture of various parts of the human body, such as Figure 3 As shown in the fifth figure in , the human body posture information may be a human body posture graph, through which the specific postures of various parts of the human body may be known.

[0085] Step B: determining a clothing area in a video frame of a human body video according to the human body key points, the human body posture information and the human body analysis result corresponding to the video frame.

[0086] Step C: masking the clothing area in the video frame to obtain a clothing-agnostic video frame.

[0087] Specifically, the preliminary clothing area in the video frame can be determined by the human body analysis results, and then the preliminary clothing area can be adjusted according to the human body key points to obtain the clothing area in the video frame. Since the human body analysis results may not mark some human details, such as hands and arms, which are a marking channel because the clothing worn by the human body is sleeveless, in this case, the area can be divided more finely through the human body key points, making the determined clothing area more accurate.

[0088] Step S130: Fusing the target clothing features with the human body features in spatial dimensions to obtain spatial fusion features.

[0089] Specifically, the coordinate system of the target clothing features and the human body features can be aligned. Specifically, the coordinates of the target clothing features can be adjusted to the same coordinate system as the human body features according to the size and proportion of the video frame. The target clothing features are mapped to the corresponding human body features and feature fusion is performed. The feature fusion method can be direct feature splicing or feature addition by weighted summation. The weighted weights can be allocated according to importance.

[0090] In a specific embodiment, Figure 5 As described above, step S130 can obtain each spatial fusion feature according to the following steps S131 to S132.

[0091] Step S131: splicing the target clothing features with the human body features corresponding to each video frame in the spatial dimension to obtain the spatial splicing features corresponding to each frame.

[0092] That is to say, the target clothing features are spliced ​​with the human body features corresponding to each frame in the spatial dimension. For example, the target clothing features are spliced ​​with the human body features corresponding to the first frame, the human body features corresponding to the second frame, and so on and so forth, respectively, in the spatial dimension to obtain the spatial splicing features corresponding to each frame.

[0093] Step S132: performing self-attention calculation on the spatial splicing features in the spatial dimension to obtain spatial fusion features.

[0094] That is, self-attention calculation is performed on the spatial splicing features corresponding to each frame to obtain the spatial fusion features corresponding to each frame, that is, the spatial fusion features corresponding to the first frame, the spatial fusion features corresponding to the second frame, and so on, the spatial fusion features corresponding to the Nth frame are obtained. Through self-attention calculation, the fusion of human body and clothing features can be achieved, thereby generating a more natural trying-on effect.

[0095] For example, for the target clothing feature (L, D) and the human body feature (L, D), where L is the feature length and D is the feature dimension, we concatenate the two features in the spatial dimension to obtain (2L, D), and then perform self-attention calculation on (2L, D) to obtain the spatial fusion feature.

[0096] Step S140: adjusting the target clothing features based on the human body features to obtain human body matching clothing features that match the human body in each video frame.

[0097] Step S140 is to adjust the target clothing features based on the human features corresponding to each frame. The obtained human matching clothing features that match the human body in the video frame can well reflect the human body features. Specifically, Figure 5 As shown, the target clothing features can be cross-attentionally calculated with each of the human features to obtain the clothing features of each human body matching the human body in each video frame. For example, the target clothing features (1, L, D) can be cross-attentionally calculated with the human body features (1, L, D), (2, L, D), ..., (T, L, D) corresponding to each frame, where T is the number of video frames, to obtain the clothing features of each human body matching (1, L ， , D ， )、(2,L ， , D ， ),……,(T,L ， , D ， ). Through the cross-attention method, human body features can be effectively combined with target clothing features quickly and easily to obtain human body matching clothing features.

[0098] Step S140 can also be adjusted in other ways to obtain human body matching clothing features. For example, the key points of the human body in the video frame can be first identified through a posture estimation model to obtain the structure and posture information of the human body, and then the features of the target clothing, including color, texture, and shape, are extracted. The extracted clothing features are mapped to the key points of the human body, and the coordinates of the clothing are adjusted according to the positions of the key points and the geometric features of the clothing to align them with the human body features. The clothing features are then scaled according to the size of the human body to adapt to the body shape of the human body, and the clothing is rotated according to the human body posture to ensure that the direction of the clothing is consistent with the human body. The adjusted clothing features are then merged with the human body features to generate the final human body matching clothing features.

[0099] Step S150: fusing each of the spatial fusion features with each of the human body matching clothing features in the time dimension to obtain a time fusion feature, and obtaining each human body fitting video frame wearing the target clothing according to the time fusion feature.

[0100] The fusion in the time dimension refers to fusing each of the spatial fusion features with each of the human body matching clothing features along the time axis to obtain a new time fusion feature.

[0101] Specifically, each of the spatial fusion features and each of the human body matching clothing features may be spliced ​​in the time dimension to obtain a time fusion feature; alternatively, the weights of each of the spatial fusion features and each of the human body matching clothing features may be determined, and the weighted sum of each of the spatial fusion features and each of the human body matching clothing features in the time dimension may be performed according to the weight to obtain the time fusion feature; alternatively, a pre-trained time dimension feature fusion model may be used to fuse each of the spatial fusion features with each of the human body matching clothing features, which is not specifically limited in the present application.

[0102] In a specific embodiment, Figure 5 As shown, step S150 can obtain the time fusion feature according to the following steps S151 to S152.

[0103] Step S151: splicing each spatial fusion feature with each body matching clothing feature in the time dimension to obtain a time splicing feature.

[0104] For example, if the spatial fusion features are (1, L ， , D ， )、(2,L ， , D ， ),……,(T,L ， , D ， ), the clothing features of each person are (1, L, D), (2, L, D), ..., (T, L, D), then the time splicing feature obtained by splicing in the time dimension is (1, L ， , D ， )、(2,L ， , D ， ),……,(T,L ， , D ， ), (T+1, L, D), (T+2, L, D), ..., (2T, L, D).

[0105] Step S152: performing self-attention calculation on the time splicing feature in the time dimension to obtain a time fusion feature.

[0106] By calculating the attention in the time dimension, the correlation between different time steps within the feature sequence can be calculated, which improves the temporal coherence of the video generated based on the time fusion feature, and ensures the consistency and accuracy of the clothing details in the video sequence. This embodiment can conveniently, efficiently and accurately fuse the spatial fusion features with the human body matching clothing features in the time dimension by splicing the features in the time dimension and performing the time dimension self-attention calculation.

[0107] In step S150, the time fusion feature corresponding to each frame can be obtained. The time fusion feature has fused the human body features of the human body video with the target clothing features in the time dimension and the space dimension. The time fusion feature is an image feature, and each human body fitting video frame can be conveniently determined based on each time fusion feature.

[0108] In a specific embodiment, step S150 can fuse the spatial fusion features with the human body matching clothing features in the time dimension to obtain the time fusion features according to the following steps S153 to S155.

[0109] Step S153: extracting the global semantic features of the target clothing from the clothing image.

[0110] The global semantic feature is used to describe the target clothing as a whole. Specifically, the global semantic feature may include the clothing style, overall structure, clothing type, clothing color, clothing pattern, etc. of the target clothing, but is not limited to these. The global semantic feature describes the overall appearance and characteristics of the target clothing. The global semantic feature can help identify and understand the category and attributes of clothing to assist in the virtual fitting task.

[0111] Specifically, the global semantic features of the target clothing can be extracted from the clothing picture through a pre-trained semantic extraction model. The training method of the semantic extraction model can adopt a supervised learning algorithm, a semi-supervised learning algorithm or an unsupervised learning algorithm, etc., which is not specifically limited in this application.

[0112] Step S154: combining the global semantic features with the spatial fusion features to obtain semantically matched spatial fusion features.

[0113] Specifically, Figure 5 The global semantic feature and the spatial fusion feature can be cross-attention calculated to combine the global semantic feature into the spatial fusion feature. The cross-attention method can make the fusion between features more complete, so that the global semantic feature can be better combined with the spatial fusion feature, so that the subsequent human fitting video frame is more consistent with the target clothing.

[0114] Alternatively, the global semantic feature may be combined with the spatial fusion feature by feature addition, feature weighted addition or feature multiplication; or, the global semantic feature may be mapped to the corresponding position of the spatial fusion feature to form a semantic matching spatial fusion feature containing the global semantic feature.

[0115] Step S155: Fusing the semantic matching fusion features with the human body matching clothing features in the time dimension to obtain a time fusion feature.

[0116] The execution process of step S155 may refer to the method of performing time dimension fusion in step S150, which will not be described in detail here.

[0117] This embodiment combines the global semantic features corresponding to the clothing image with the spatial fusion features, that is, combines the global semantic features with the human body features corresponding to the video frame. In this way, the subsequent temporal fusion features combine the target clothing features, the global semantics of the target clothing, and the consistency in the temporal dimension at the same time, so that the final human body fitting video frame can be more consistent with the target clothing in overall style, thereby achieving an in-depth understanding of clothing features in different spatial dimensions and enhancing the retention of the clothing style and structure of the target clothing.

[0118] In a specific embodiment, the three steps of feature fusion, step S130, step S154, and step S155, can be implemented according to the following step S160. That is, the three steps of feature fusion, step S130, step S154, and step S155, can be implemented simultaneously through step S160.

[0119] Step S160: Perform feature processing on the human body features corresponding to each frame through each feature processing layer of the pre-trained image generation model to obtain processed human body features corresponding to each frame that are fused with the target clothing features, fused with the global semantic features, and keep the features of each frame consistent in time series.

[0120] The human body features after the above processing may be a human body fitting video frame, or may be feature data reflecting the human body fitting video frame.

[0121] The above-mentioned image generation model is used to generate time fusion features reflecting the human body fitting video frame according to the input human body features, target clothing features and global semantic features, and obtain the corresponding human body fitting video frame. The image generation model is generated in advance, and the specific training process will be described later. The image generation model can be a diffusion model, specifically a stable diffusion model (Stable Diffusion, referred to as SD), a latent diffusion model (Latent Diffusion Model, referred to as LDM), etc., or other diffusion models. The image generation model can also be other models, such as a neural network model, a generative adversarial network, etc., which are not specifically limited in this application. Among them, the diffusion model can restore a clear image from the noise through a progressive denoising process. Through multiple iterative denoising steps, the diffusion model can capture the complex structure and details in the image, thereby generating a high-resolution, high-quality output. Therefore, in the embodiment of the present application, the image generation model can be a diffusion model. Correspondingly, the clothing encoding model can also be a diffusion model of the same structure, so that the clothing features output by each layer of the clothing encoding model are fused with the corresponding feature processing layer of the image generation model.

[0122] The input of the i-th feature processing layer of the image generation model is obtained through the following steps S161 to S163.

[0123] Step S161: Fusing the output of the i-1th feature processing layer corresponding to each frame with the target clothing feature in the spatial dimension to obtain the i-1th spatial fusion feature corresponding to each frame.

[0124] Among them, the output of the 0th feature processing layer is the human body feature, and the output of the 0th feature processing layer is the input information of the image generation model, such as Figure 4 The input information of the image generation model may include each of the clothing agnostic video frames, each of the masks, and each of the human body analysis results. Since the human body features corresponding to each frame are input into the image generation model, the outputs of each feature processing layer of the image generation model are also the outputs corresponding to each frame. The way in which step S161 performs fusion in the spatial dimension is similar to the way in which step S130 performs feature fusion, and how to perform feature fusion in the spatial dimension will not be described in detail here.

[0125] Step S162: combining the global semantic feature with the i-1th spatial fusion feature to obtain the i-1th semantic matching spatial fusion feature corresponding to each frame.

[0126] The process of combining the global semantic feature with the (i-1)th spatial fusion feature in step S162 may refer to the process of combining the global semantic feature with the spatial fusion feature in step S154, which will not be described in detail here.

[0127] Step S163: Fuse the i-1th semantic matching space fusion feature corresponding to each frame with each of the human body matching clothing features in the time dimension to obtain the i-1th time fusion feature, and determine the i-1th time fusion feature as the input of the i-th feature processing layer.

[0128] Here, i traverses 1 to N, and N is the number of feature processing layers included in the image generation model. The way of fusing in the time dimension in step S163 can refer to the way of fusing in the time dimension in step S150, which will not be described in detail here.

[0129] Exemplarily, the process of generating human fitting video frames through an image generation model is as follows: first, the human features corresponding to each frame are fused with the target clothing features in the spatial dimension to obtain the 0th spatial fusion feature corresponding to each frame, the global semantic features are combined with the 0th spatial fusion features to obtain the 0th semantic matching spatial fusion features corresponding to each frame, the 0th semantic matching spatial fusion features corresponding to each frame are fused with each human matching clothing feature in the time dimension to obtain the 0th time fusion feature, and the 0th time fusion feature is determined as the input of the first feature processing layer, so that the first feature processing layer obtains the corresponding output after feature processing, and then the output of the first feature processing layer corresponding to each frame is fused with the target clothing features in the spatial dimension to obtain The first spatial fusion feature corresponding to each frame, the global semantic feature is combined with the first spatial fusion feature to obtain the first semantic matching spatial fusion feature corresponding to each frame, and then the first semantic matching spatial fusion feature corresponding to each frame is fused with each human body matching clothing feature in the time dimension to obtain the first time fusion feature, and the first time fusion feature is determined as the input of the second feature processing layer, so that the second feature processing layer obtains the corresponding output after feature processing, ..., until the N-th feature processing layer obtains the corresponding output after feature processing, wherein the output of the N-th feature processing layer is the final processed human body feature corresponding to each frame that is fused with the target clothing feature, the global semantic feature and the features of each frame that remain consistent in the time series.

[0130] This embodiment uses a pre-trained image generation model and controls feature fusion when the model performs feature processing, so as to more accurately generate processed human body features corresponding to each frame that are fused with target clothing features, fused with the global semantic features, and keep the features of each frame consistent in time series, thereby quickly, efficiently and accurately obtaining each frame of the human body fitting video frame.

[0131] In a specific embodiment, Figure 4The clothing coding model may include multiple clothing coding layers. Step S153 may extract the global semantic features of the target clothing according to the following step S153a.

[0132] Step S153a: extracting global semantic features of the target clothing from the clothing image through each visual extraction layer of the pre-trained visual model.

[0133] The visual model is used to extract the global semantic features of the target clothing from the clothing picture through various visual extraction layers. The visual model can be a neural network model, a DINOv2 encoder, etc., or other models that can extract the global semantics of the picture. Among them, the DINOv2 encoder is a self-supervised pre-trained visual model.

[0134] Step S161 can obtain the i-1th spatial fusion feature corresponding to each frame according to the following step S161a.

[0135] Step S161a: Fuse the output of the i-1th feature processing layer corresponding to each frame with the i-1th layer of target clothing features in the spatial dimension to obtain the i-1th spatial fusion feature corresponding to each frame.

[0136] The target clothing features of the i-1th layer are features output by the clothing coding layer of the i-1th layer of the clothing coding model.

[0137] Step S162 can obtain the i-1th semantic matching space fusion feature corresponding to each frame according to the following step S162a.

[0138] Step S162a: Fusing the i-1th spatial fusion feature with the i-1th layer semantic feature to obtain the i-1th semantic matching spatial fusion feature corresponding to each frame.

[0139] The i-1th layer semantic features are features output by the i-1th layer visual extraction layer of the visual model.

[0140] Exemplarily, when the i-1th spatial fusion feature is fused with the i-1th layer semantic feature by cross attention, for the i-1th layer network, the i-1th semantic matching spatial fusion feature h i-1 =Attention(q i-1 ,k i-1 ,v i-1 ), where q i-1 is the feature map of the i-1th layer of the image generation model (i.e., the output of the i-1th layer), k i-1 ,v i-1 It is the feature map of the i-1th layer of the visual model.

[0141] When performing feature fusion, this embodiment fuses the output features of the corresponding feature processing layers of each model in a layered one-to-one correspondence, which can achieve an in-depth understanding of clothing features and clothing semantics in different spatial dimensions, so that the obtained human body video frames can better show the try-on effect.

[0142] In a specific embodiment, the human body analysis result can be represented in the form of a human body analysis result graph. Figure 4 The step S160 may perform feature processing on the human body features corresponding to each frame according to the following steps S164 to S165.

[0143] Step S164: extracting features from the human body posture graph using a pre-trained image feature extraction model to obtain human body posture features.

[0144] Step S165: Determine the model input of the pre-trained image generation model according to the human posture features, the clothing agnostic video frame, and the mask, and input the model into the image generation model to perform feature processing on the human features corresponding to each frame through each feature processing layer of the image generation model.

[0145] Step S165 may determine the human body posture features, the clothing agnostic video frame, and the mask as model inputs of an image generation model.

[0146] Alternatively, other information may be added as model input. For example, when the image generation model is a diffusion model, Figure 4 In step S165, the model input of the pre-trained image generation model can be determined in the following manner: obtaining random noise; determining the model input of the pre-trained image generation model according to the random noise, the video frame of the human body video, the human body posture feature, the clothing agnostic video frame, and the mask, wherein the random noise is used to add noise to the video frame. In this embodiment, when the image generation model is a diffusion model, random noise can be input so that the diffusion model can add noise to the video frame of the human body video, and then gradually denoising is performed to obtain the human body video frame.

[0147] Specifically, Figure 5The pre-trained image generation model may be a pre-trained denoising network U-Net, and the model input of the pre-trained image generation model is determined according to the random noise, the video frame of the human body video, the human body posture feature, the clothing agnostic video frame, and the mask, which may be implemented according to the following steps: adding noise to the video frame of the human body video by the random noise to obtain a noisy video frame, splicing the noisy video frame, the clothing agnostic video frame, and the mask to obtain a spliced ​​input feature, and then adding the spliced ​​input feature to the human body posture feature to obtain an added input feature, and determining the added input feature as the input information of the denoising network U-Net.

[0148] In a specific embodiment, before step S120, the method may further include the following step S120a.

[0149] Step S120a: Divide the human body video into a plurality of video segments, wherein there are overlapping video frames between two adjacent video segments.

[0150] In the embodiment of the present application, the number of video frames contained in each video segment may be the same or different. In order to facilitate calculation and make the calculation result more accurate, the number of video frames contained in each video segment may be the same. The number of video frames contained in each video segment can be determined according to the computing power of the electronic device. If the computing power is strong, the number of video frames contained can be more. If the computing power is low, the number of video frames contained can be less. Exemplarily, taking a 28-frame human body video as an example, it can be cut into two 16-frame video segments, with 4 overlapping frames in the middle.

[0151] Accordingly, the virtual try-on video generation method can generate a human try-on video frame for each video segment, and obtain a human try-on video frame corresponding to each video frame. Specifically, the electronic device can generate a human try-on video frame for each video segment in order from front to back, and obtain a human try-on video frame corresponding to each video frame.

[0152] For the overlapping video frame, the human body try-on video frame corresponding to the overlapping video frame is determined according to the previous human body try-on video frame and the subsequent human body try-on video frame corresponding to the overlapping video frame, the previous human body try-on video frame is the human body try-on video frame corresponding to the overlapping video frame calculated for the previous video segment to which the overlapping video frame belongs, and the subsequent human body try-on video frame is the human body try-on video frame corresponding to the overlapping video frame calculated for the subsequent video segment to which the overlapping video frame belongs.

[0153] Specifically, the average data of the previous human fitting video frame and the next human fitting video frame corresponding to the overlapping video frame can be determined as the human fitting video frame corresponding to the overlapping video frame, or the weighted average value of the previous human fitting video frame and the next human fitting video frame corresponding to the overlapping video frame can be calculated to obtain the human fitting video frame corresponding to the overlapping video frame. Exemplarily, taking a 28-frame video as an example, it is cut into two 16-frame video clips with 4 overlapping frames in the middle, and the two clips are inferred respectively, and the average result of inference is used for the middle 4 frames to achieve a smooth transition of the video.

[0154] This embodiment calculates the fitting video frames corresponding to the overlapping video frames through the sliding window method. Due to the limitation of computing resources of electronic devices, it is difficult to infer a long video at one time. In order to process long videos, this solution adopts the sliding window method to enhance the temporal coherence and smoothness of long videos. Since there are overlaps between the segments, feature transfer can be achieved to ensure a smooth transition between the segments. In this process, the average value of the inference results of all overlapping frames is taken as the final processing result of these frames, which not only makes full use of the information of each frame, but also ensures the smooth transition of dynamic continuity in the video and the consistency of the inference results.

[0155] The virtual try-on video generation method provided in the embodiment of the present application can be named as the Generative Preserving Diffusion for Video Virtual Try-On (GPD-VVTO for short) method. The solution provided in the embodiment of the present application generates human try-on video frames through a diffusion model.

[0156] Each step in the embodiment of the present application can be implemented by a joint self-attention (JSA) module, a semantic-enhanced cross-attention (SCA) module, and a garment-aware temporal attention (SCA) module. Attention, referred to as GTA) module, three modules work in coordination, wherein the JSA module is used to execute step S130 to fuse the target clothing features with each of the human body features in the spatial dimension to obtain each spatial fusion feature, that is, to execute step S161 to fuse the output of the i-1th feature processing layer corresponding to each frame with the target clothing features in the spatial dimension to obtain the i-1th spatial fusion feature corresponding to each frame, the SCA module is used to execute step S154 to combine the global semantic features with the spatial fusion features to obtain each semantic matching spatial fusion feature, that is, to execute step S162 to combine the global semantic features with the i-1th spatial fusion features to obtain the i-1th semantic matching spatial fusion feature corresponding to each frame, the GTA module is used to execute step S140 to adjust the target clothing features based on each of the human body features to obtain each human body matching clothing feature matching the human body in each video frame, and step S155 to fuse each of the semantic matching fusion features with each of the human body matching clothing features in the time dimension to obtain the time fusion feature step, such as Figure 4 The schematic diagram shows the data processing process of each module in each layer of the network. Through these three modules, the clothing features can be combined with the human body features in space, the clothing tried on by the human body features can be matched with the target clothing in semantics, and the try-on video can maintain frame continuity in the time dimension.

[0157] The virtual try-on video generation method provided in the embodiment of the present application obtains a human body video to be virtually tried on and a clothing picture corresponding to a target clothing, extracts target clothing features corresponding to the target clothing from the clothing picture, and extracts human body features corresponding to each video frame from the above human body video, and fuses the target clothing features with each human body feature in the spatial dimension to obtain spatial fusion features corresponding to each video frame. The spatial fusion features can well represent the features after the human body features in each video frame are combined with the target clothing features, so as to represent the basic appearance display of the target clothing being tried on the human body in the video frame of the human body video, and then adjusts the target clothing features based on each human body feature to obtain each human body matching clothing feature that matches the human body in each video frame. The human body matching clothing feature can match the human body posture and human body structure in the corresponding video frame, so as to make the virtual try-on more effective. The result is more in line with the human body state. For example, when the human body is twisting its body posture, the human body matching clothing feature is the state feature of the target clothing when the human body is twisting its body posture. Then, each spatial fusion feature is fused with each human body matching clothing feature in the time dimension. Since each spatial fusion feature is the feature after each video frame of the human body video is fused with the target clothing feature, therefore, each spatial fusion feature is fused with each human body matching clothing feature in the time dimension, that is, the target clothing feature is indirectly fused with the human body feature of each video frame in the time dimension, which can better ensure that the target clothing feature is consistent with each frame of the human body video in the time dimension, so that the obtained time fusion feature corresponding to each frame can not only well reflect the effect of a single video frame trying on the target clothing, but also make each human body trying on the target clothing obtained according to the above time fusion feature can also maintain the consistency and consistency of each frame in the time dimension.

[0158] It can be seen that the solution provided by the present application can well generate the effect of trying on the target clothing in a single video frame by fusing the target clothing features with the human body features in the spatial dimension, and can achieve accurate retention of clothing details. By fusing each spatial fusion feature with each human body matching clothing feature in the temporal dimension, the coherence and consistency of each frame can be maintained in the temporal dimension, so that the generated virtual try-on video has better inter-frame continuity and smoothness, thereby improving the authenticity of the final virtual try-on video and achieving a better virtual try-on effect. The present application not only improves the generation quality of single-frame images, but also ensures the natural transition and true performance of clothing in the entire video sequence, thereby providing users with a more realistic and coherent virtual try-on experience.

[0159] Embodiment 2

[0160] The second embodiment of the present application also provides a model training method, which is applied to electronic devices, which can be electronic devices with data processing capabilities such as desktop computers, laptops, servers, mobile terminals, gateways, etc. This embodiment is at least used to train the image generation model required for virtual try-on video generation in the first embodiment. Other models in the first embodiment (such as clothing coding models, visual models, image feature extraction models, etc.) can be trained during the image generation model training process, or can be obtained by pre-training. The model training method of this embodiment will synchronously train the training processes of the image generation model, clothing coding model, visual model, and image feature extraction model.

[0161] like Figure 6 As shown, the model training method provided in this embodiment includes the following steps S210 to S250.

[0162] Step S210: Obtain training samples.

[0163] The training samples include sample human videos, sample clothing pictures corresponding to sample target clothing, and sample fitting videos of the sample target clothing being fitted. In the present application, a large number of training samples can be obtained to accurately train the model. The resolution of the videos in the training samples is usually high, and a large number of training samples include sample clothing pictures corresponding to a variety of sample target clothing, a large number of different types of human videos, etc., to provide rich training resources for the training and evaluation of the model.

[0164] Step S220: extracting sample target clothing features corresponding to the sample target clothing from the sample clothing picture, and extracting sample human body features corresponding to each sample video frame from the sample human body video.

[0165] Step S230: adjusting the sample target clothing features based on the sample human features to obtain matching clothing features of each sample human body that matches the human body in each sample video frame.

[0166] Step S240: Perform feature processing on the sample human body features corresponding to each frame through each feature processing layer of the image generation model to be trained, and obtain output processed human body features corresponding to each frame that are integrated with the sample target clothing features and keep the features of each frame consistent in the time series, and determine each output human body fitting video frame trying on the sample target clothing based on the output processed human body features.

[0167] The execution process of step S220 can refer to the execution process of step S120 in the first embodiment, and the execution process of step S230 can refer to the execution process of step S140 in the first embodiment. In step S240, the sample human body features corresponding to each frame are feature processed by each feature processing layer of the image generation model to be trained to obtain the output processed human body features corresponding to each frame that are fused with the sample target clothing features and the features of each frame remain consistent in the time series. The execution process can refer to step S160 in the first embodiment. The step of determining each output human body fitting video frame trying on the sample target clothing according to the output processed human body features in step S240 can refer to the step of obtaining each human body fitting video frame trying on the target clothing according to the time fusion features in step S150 in the first embodiment, which will not be described in detail here.

[0168] Step S250: Based on the consistency between the output human fitting video frame and each sample video frame of the sample fitting video, adjusting the parameters of the image generation model to be trained to obtain a trained image generation model.

[0169] Step S250 uses a supervised training algorithm to adjust the parameters of the training image generation model. Specifically, the gradient descent method or other optimization algorithms can be used to update the model parameters through back propagation. The specific parameter adjustment method can use the supervised training algorithm of related technology, which will not be described in detail here.

[0170] Among them, during the model training process, the input of the i-th feature processing layer of the image generation model to be trained is obtained through the following steps S241 to S242.

[0171] Step S241: Fusing the output of the i-1th feature processing layer corresponding to each sample frame with the sample target clothing feature in the spatial dimension to obtain the i-1th spatial fusion feature of the sample corresponding to each sample frame.

[0172] Among them, the output of the 0th feature processing layer is the sample human body feature, and the output of the 0th feature processing layer is the input of the image generation model to be trained.

[0173] Step S242: Fuse the i-1th spatial fusion feature of the sample corresponding to each sample frame with the human body matching clothing feature of each sample in the time dimension to obtain the i-1th time fusion feature of the sample, and determine the i-1th time fusion feature of the sample as the input of the i-th feature processing layer.

[0174] i traverses 1 to N, where N is the number of feature processing layers included in the image generation model to be trained.

[0175] The way of fusing in the spatial dimension of step S241 can refer to step S161 in the first embodiment, and the way of fusing in the temporal dimension of step S242 can refer to step S163 in the first embodiment, which will not be described in detail in this embodiment.

[0176] In one implementation, before step S240, the model training method may further include the following step S240a.

[0177] Step S240a: extracting sample global semantic features of the sample target clothing from the sample clothing image.

[0178] Step S240 can be implemented by following the steps S240-1.

[0179] Step S240-1: Perform feature processing on the sample human features corresponding to each frame through each feature processing layer of the image generation model to be trained, and obtain output processed human features corresponding to each frame that are fused with the sample target clothing features, fused with the sample global semantic features, and keep the features of each frame consistent in the time series.

[0180] The execution process of step S240 - 1 may refer to step S160 in the first embodiment.

[0181] Step S242 can obtain the i-1th time fusion feature of the sample according to the following steps S242a to S242b.

[0182] Step S242a: combining the sample global semantic features with the sample i-1th spatial fusion features to obtain the sample i-1th semantic matching spatial fusion features corresponding to each sample frame.

[0183] Step S242b: Fusing the sample i-1th semantic matching spatial fusion feature corresponding to each sample frame with each human body matching clothing feature in the time dimension to obtain the sample i-1th time fusion feature.

[0184] The execution process of step S242a may refer to step S162 in the first embodiment, and the execution process of step S242b may refer to step S163 in the first embodiment, which will not be described in detail in this embodiment.

[0185] In this embodiment, since the image generation model is controlled to incorporate the global semantic features of the sample target clothing during the model training process, the trained image generation model can generate a human body fitting video that better matches the overall semantics of the target clothing.

[0186] In one implementation, step S220 may extract sample target clothing features corresponding to the sample target clothing from the sample clothing image according to the following step S221.

[0187] Step S221: performing clothing encoding on the sample clothing image through the clothing encoding model to be trained to obtain sample target clothing features corresponding to the sample target clothing, wherein the clothing encoding model to be trained includes multiple clothing encoding layers to be trained.

[0188] Step S241 can obtain the i-1th spatial fusion feature of the sample corresponding to each sample frame according to the following step S241a.

[0189] Step S241a: Fuse the output of the i-1th feature processing layer corresponding to each sample frame with the i-1th layer target clothing feature of the sample in the spatial dimension to obtain the i-1th spatial fusion feature of the sample corresponding to each sample frame.

[0190] The target clothing features of the sample at the i-1th layer are features outputted by the clothing coding layer at the i-1th layer to be trained of the clothing coding model to be trained.

[0191] The model training method may further include the following step S260.

[0192] Step S260: Based on the consistency between the output human fitting video frame and each sample video frame of the sample fitting video, the parameters of the clothing coding model to be trained are adjusted to obtain a trained clothing coding model.

[0193] The parameter adjustment method of step S260 is similar to the parameter adjustment method of step S260, which will not be described in detail here. This embodiment can simultaneously train the clothing coding model, so that the image of the target clothing can be encoded more accurately, thereby improving the accuracy of virtual dressing.

[0194] In one implementation, step S240a may extract the sample global semantic features of the sample target clothing from the sample clothing image according to the following step S240a-1.

[0195] Step S240a-1: extracting sample global semantic features of the sample target clothing from the sample clothing image through each visual extraction layer of the visual model to be trained.

[0196] Step S242a can obtain the i-1th semantic matching spatial fusion feature of the sample corresponding to each sample frame according to the following step S242a-1.

[0197] Step S242a-1: Fuse the i-1th spatial fusion feature of the sample with the i-1th layer semantic feature of the sample to obtain the i-1th semantic matching spatial fusion feature of the sample corresponding to each sample frame, wherein the i-1th layer semantic feature of the sample is the feature output by the i-1th layer visual extraction layer of the sample of the visual model to be trained.

[0198] The model training method may further include the following step S270.

[0199] Step S270: Based on the consistency between the output human fitting video frame and each sample video frame of the sample fitting video, the parameters of the visual model to be trained are adjusted to obtain a trained visual model.

[0200] This implementation can simultaneously train the visual model, thereby improving the training efficiency, and can also enable more accurate semantic extraction of target clothing images, thereby improving the accuracy of virtual dressing.

[0201] In a specific embodiment, the image generation model to be trained may be a diffusion model to be trained, and may be a stable diffusion model or a potential diffusion model, or other diffusion models.

[0202] Correspondingly, the input of the diffusion model to be trained may also include preset noise, and the diffusion model to be trained is also used to output predicted noise. In other words, the input of the diffusion model to be trained also includes the preset noise.

[0203] The model training method may further include the following step S280.

[0204] Step S280: Based on the consistency between the predicted noise and the preset noise, the parameters of the image generation model to be trained are adjusted to obtain a trained image generation model.

[0205] This embodiment can obtain a more accurate image generation model by training the diffusion model.

[0206] This embodiment introduces the training process of the estimation model from the perspective of model training. The processing process of the training samples by the to-be-trained model during the training process is similar to the processing process of clothing pictures and human body videos by the image generation model during the virtual try-on video generation process. Therefore, the relevant content and beneficial effects of the specific implementation process of this embodiment can be referred to the first embodiment, and the similarities will not be described in detail in this embodiment.

[0207] The model training method provided in the present application obtains training samples, and the training samples include sample human body videos, sample clothing pictures corresponding to sample target clothing, and sample fitting videos of the above-mentioned sample target clothing being tried on, extracts sample target clothing features corresponding to the sample target clothing from the sample clothing pictures, and extracts sample human body features corresponding to each sample video frame from the sample human body video; performs feature adjustment on the sample target clothing features based on each sample human body feature, and obtains each sample human body matching clothing feature that matches the human body in each sample video frame; performs feature processing on the sample human body features corresponding to each frame through each feature processing layer of the image generation model to be trained, and obtains output processed human body features corresponding to each frame that are fused with the sample target clothing features and whose features are consistent in time series, and determines each output human body fitting video of trying on the sample target clothing based on the output processed human body features. frequency frames; wherein, the input of the i-th feature processing layer of the image generation model to be trained is obtained by the following steps: the output of the i-1th feature processing layer corresponding to each sample frame is fused with the sample target clothing feature in the spatial dimension to obtain the sample i-1th spatial fusion feature corresponding to each sample frame; the sample i-1th spatial fusion feature corresponding to each sample frame is fused with the sample body matching clothing feature in the time dimension to obtain the sample i-1th time fusion feature, and the sample i-1th time fusion feature is determined as the input of the i-th feature processing layer, i traverses 1 to N, N is the number of feature processing layers included in the image generation model to be trained, and the output of the 0th feature processing layer is the sample body feature; based on the consistency between the output human body fitting video frame and each sample video frame of the sample fitting video, the parameters of the image generation model to be trained are adjusted to obtain the trained image generation model. In the process of training the image generation model of the present application, the control model can obtain the output processed human body features corresponding to each frame that are integrated with the sample target clothing features and the features of each frame are consistent in the time series, so that the trained image generation model can not only well guarantee the effect of trying on the target clothing of the generated single video frame, but also enable the video frames generated by the image generation model to maintain the coherence and consistency of each frame in the time dimension, so that the virtual try-on video generated by the image generation model trained by the method provided by the present application has better inter-frame continuity and smoothness, thereby improving the authenticity of the final virtual try-on video and achieving a better virtual try-on effect.

[0208] Embodiment 3

[0209] The third embodiment of the present application also provides a model training method, which is applied to an electronic device, which may be a desktop computer, a laptop computer, a server, a mobile terminal, a gateway, or other electronic device with data processing capabilities.

[0210] like Figure 7As shown, the model training method provided in this embodiment includes the following steps S310 to S370.

[0211] Step S310: obtaining training samples, wherein the training samples include sample human body videos, sample clothing pictures corresponding to sample target clothing, and sample fitting videos of people trying on the sample target clothing;

[0212] Step S320: extracting sample target clothing features corresponding to the sample target clothing from the sample clothing picture, and extracting sample human body features corresponding to each sample video frame from the sample human body video;

[0213] Step S330: feature processing is performed on the sample human body features corresponding to each frame through each feature processing layer of the image generation model to be trained, and the first output processed human body features fused with the sample target clothing features are obtained, and each first output human body fitting video frame fitting with the sample target clothing is determined according to the first output processed human body features; wherein the input of the i-th feature processing layer of the image generation model to be trained is obtained by the following steps: the output of the i-1th feature processing layer corresponding to each sample frame is fused with the sample target clothing features in the spatial dimension to obtain the i-1th spatial fusion features of the samples corresponding to each sample frame, and the input of the i-th feature processing layer is determined according to the i-1th spatial fusion features, i traverses 1 to N, N is the number of feature processing layers included in the image generation model to be trained, and the output of the 0th feature processing layer is the sample human body features;

[0214] Step S340: Based on the consistency between the first output human fitting video frame and each sample video frame of the sample fitting video, adjusting the parameters of the image generation model to be trained to obtain the image generation model after first-stage training;

[0215] Step S350: adjusting the sample target clothing features based on the sample human body features to obtain matching clothing features of each sample human body that matches the human body in each sample video frame;

[0216] Step S360: Perform feature processing on the sample human features corresponding to each frame through each feature processing layer of the image generation model after training in the first stage, obtain the second output processed human features corresponding to each frame which are fused with the sample target clothing features and the features of each frame are consistent in the time series, and determine each second output human fitting video frame trying on the sample target clothing according to the second output processed human features; wherein the input of the i-th feature processing layer of the image generation model after training in the first stage is obtained by the following steps: fuse the output of the i-1th feature processing layer corresponding to each sample frame with the sample target clothing features in the spatial dimension to obtain the sample i-1th spatial fusion feature corresponding to each sample frame; fuse the sample i-1th spatial fusion feature corresponding to each sample frame with each sample human matching clothing feature in the time dimension to obtain the sample i-1th time fusion feature, and determine the sample i-1th time fusion feature as the input of the i-th feature processing layer, i traverses 1 to N, N is the number of feature processing layers included in the image generation model to be trained, and the output of the 0th feature processing layer is the sample human features;

[0217] Step S370: Based on the consistency between the second output human fitting video frame and each sample video frame of the sample fitting video, the parameters of the first stage trained image generation model are adjusted to obtain a trained image generation model.

[0218] This embodiment trains the model in two stages. The first stage is trained on a single-frame image. In this stage, the image generation model to be trained can be initialized as a 2D convolution. The first stage is the image generation model after the first stage training obtained by step S310 to step S340. The image generation model after the first stage training can generate a human body fitting video frame with high single-frame accuracy. The second stage is the image generation model obtained by step S350 to step S370. In the second stage, the image generation model can be expanded to 3D convolution. The image generation model obtained by the second stage training can generate a human body fitting video frame with high single-frame accuracy and ensure the continuity between each video frame. The accuracy of model training can be improved by the two-stage training method.

[0219] In a specific embodiment, when the image generation model to be trained is a diffusion model, the model training method can be referred to the previous embodiment, and the model parameters can be adjusted in both stages by predicting the consistency between the noise and the input noise.

[0220] In the process of model training in this embodiment, except for performing model training in stages, other training processes are similar to the model training process in the second embodiment and will not be described in detail here.

[0221] Embodiment 4

[0222] The fourth embodiment of the present application also provides a virtual try-on video generation device corresponding to the virtual try-on video generation method embodiment provided in the first embodiment. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple. For the details of the relevant technical features and the effects achieved, please refer to the corresponding description of the virtual try-on video generation method embodiment provided above. The virtual try-on video generation device provided in this embodiment includes:

[0223] An acquisition unit, used to acquire a human body video to be virtually tried on and a clothing picture corresponding to the target clothing;

[0224] An extraction unit, configured to extract target clothing features corresponding to the target clothing from the clothing picture, and to extract human features corresponding to each video frame from the human body video;

[0225] A spatial fusion unit, used to fuse the target clothing features with the human body features in the spatial dimension to obtain spatial fusion features;

[0226] An adjustment unit, configured to perform feature adjustment on the target clothing features based on the human body features to obtain human body matching clothing features that match the human body in each video frame;

[0227] The time fusion unit is used to fuse each of the spatial fusion features with each of the human body matching clothing features in the time dimension to obtain a time fusion feature, and obtain each human body fitting video frame wearing the target clothing according to the time fusion feature.

[0228] Embodiment 5

[0229] The fifth embodiment of the present application also provides a model training device corresponding to the model training method embodiment provided in the second embodiment. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple. For the details of the relevant technical features and the effects achieved, please refer to the corresponding description of the model training method embodiment provided above. The model training device provided in this embodiment includes:

[0230] A sample acquisition unit, used to acquire training samples, wherein the training samples include sample human body videos, sample clothing pictures corresponding to sample target clothing, and sample fitting videos of people trying on the sample target clothing;

[0231] A sample extraction unit, configured to extract sample target clothing features corresponding to the sample target clothing from the sample clothing picture, and to extract sample human body features corresponding to each sample video frame from the sample human body video;

[0232] A sample adjustment unit, configured to adjust the sample target clothing features based on the sample human features to obtain matching clothing features of each sample human body that matches the human body in each sample video frame;

[0233] A sample fusion unit is used to perform feature processing on the sample human body features corresponding to each frame through each feature processing layer of the image generation model to be trained, so as to obtain output processed human body features corresponding to each frame which are fused with the sample target clothing features and the features of each frame are kept consistent in time series, and to determine each output human body fitting video frame fitting the sample target clothing according to the output processed human body features;

[0234] A parameter adjustment unit, configured to adjust the parameters of the image generation model to be trained based on the consistency between the output human fitting video frame and each sample video frame of the sample fitting video, so as to obtain a trained image generation model;

[0235] The input of the i-th feature processing layer of the image generation model to be trained is obtained by the following steps:

[0236] The output of the i-1th feature processing layer corresponding to each sample frame is fused with the sample target clothing feature in the spatial dimension to obtain the i-1th spatial fusion feature of the sample corresponding to each sample frame;

[0237] The i-1th spatial fusion feature of the sample corresponding to each sample frame is fused with the clothing feature matching each sample body in the time dimension to obtain the i-1th time fusion feature of the sample, and the i-1th time fusion feature of the sample is determined as the input of the i-th feature processing layer, i traverses 1 to N, N is the number of feature processing layers included in the image generation model to be trained, and the output of the 0th feature processing layer is the sample body feature.

[0238] The sixth embodiment of the present application also provides an electronic device embodiment corresponding to the virtual try-on video generation method provided in the first embodiment. The following description of the electronic device embodiment is only illustrative. The electronic device embodiment is as follows:

[0239] Please refer to Figure 8 Understand the above electronic devices, Figure 8 The electronic device provided in this embodiment includes: a processor 1001, a memory 1002, a communication bus 1003, and a communication interface 1004;

[0240] The memory 1002 is used to store computer instructions for data processing. When the computer instructions are read and executed by the processor 1001, the following steps are performed:

[0241] Obtaining a human body video to be virtually tried on and a clothing picture corresponding to the target clothing;

[0242] Extracting target clothing features corresponding to the target clothing from the clothing picture, and extracting human body features corresponding to each video frame from the human body video;

[0243] The target clothing features are respectively fused with the human body features in the spatial dimension to obtain spatial fusion features;

[0244] Based on the human body features, the target clothing features are adjusted to obtain human body matching clothing features that match the human body in each video frame;

[0245] Each of the spatial fusion features is fused with each of the human body matching clothing features in the time dimension to obtain a time fusion feature, and each human body fitting video frame wearing the target clothing is obtained based on the time fusion feature.

[0246] The seventh embodiment of the present application also provides an electronic device embodiment corresponding to the model training method provided in the second embodiment. The following description of the electronic device embodiment is only illustrative. The electronic device embodiment is as follows:

[0247] The electronic device provided in this embodiment includes: a processor, a memory, a communication bus, and a communication interface;

[0248] The memory is used to store computer instructions for data processing, and when the computer instructions are read and executed by the processor, the following steps are performed:

[0249] Acquire training samples, wherein the training samples include a sample human body video, a sample clothing picture corresponding to a sample target clothing, and a sample fitting video of a person fitting the sample target clothing;

[0250] Extracting sample target clothing features corresponding to the sample target clothing from the sample clothing picture, and extracting sample human body features corresponding to each sample video frame from the sample human body video;

[0251] Based on each of the sample human body features, feature adjustment is performed on the sample target clothing features to obtain each sample human body matching clothing feature that matches the human body in each sample video frame;

[0252] Performing feature processing on the sample human body features corresponding to each frame through each feature processing layer of the image generation model to be trained, obtaining output processed human body features corresponding to each frame that are integrated with the sample target clothing features and the features of each frame are consistent in time series, and determining each output human body fitting video frame fitting the sample target clothing according to the output processed human body features;

[0253] Based on the consistency between the output human fitting video frame and each sample video frame of the sample fitting video, adjusting the parameters of the image generation model to be trained to obtain a trained image generation model;

[0254] The input of the i-th feature processing layer of the image generation model to be trained is obtained by the following steps:

[0255] The output of the i-1th feature processing layer corresponding to each sample frame is fused with the sample target clothing feature in the spatial dimension to obtain the i-1th spatial fusion feature of the sample corresponding to each sample frame;

[0256] The i-1th spatial fusion feature of the sample corresponding to each sample frame is fused with the clothing feature matching each sample body in the time dimension to obtain the i-1th time fusion feature of the sample, and the i-1th time fusion feature of the sample is determined as the input of the i-th feature processing layer, i traverses 1 to N, N is the number of feature processing layers included in the image generation model to be trained, and the output of the 0th feature processing layer is the sample body feature.

[0257] The eighth embodiment of the present application also provides a computer-readable storage medium for implementing the method described in the first embodiment. The computer-readable storage medium embodiment provided in the present application is described relatively simply. For the relevant parts, please refer to the corresponding description of the above method embodiment. The following described embodiment is only illustrative.

[0258] The computer readable storage medium provided in this embodiment stores computer instructions, and when the instructions are executed by a processor, the following steps are implemented:

[0259] Obtaining a human body video to be virtually tried on and a clothing picture corresponding to the target clothing;

[0260] Extracting target clothing features corresponding to the target clothing from the clothing picture, and extracting human body features corresponding to each video frame from the human body video;

[0261] The target clothing features are respectively fused with the human body features in the spatial dimension to obtain spatial fusion features;

[0262] Based on the human body features, the target clothing features are adjusted to obtain human body matching clothing features that match the human body in each video frame;

[0263] Each of the spatial fusion features is fused with each of the human body matching clothing features in the time dimension to obtain a time fusion feature, and each human body fitting video frame wearing the target clothing is obtained based on the time fusion feature.

[0264] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0265] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0266] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0267] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0268] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A method for generating a virtual try-on video, characterized in that: The method comprises: Obtaining a human body video to be virtually tried on and a clothing picture corresponding to the target clothing; Extracting target clothing features corresponding to the target clothing from the clothing picture, and extracting human body features corresponding to each video frame from the human body video; The target clothing features are respectively fused with the human body features in the spatial dimension to obtain spatial fusion features; Based on the human body features, the target clothing features are adjusted to obtain human body matching clothing features that match the human body in each video frame; Each of the spatial fusion features is fused with each of the human body matching clothing features in the time dimension to obtain a time fusion feature, and each human body fitting video frame wearing the target clothing is obtained based on the time fusion feature.

2. The method for generating a virtual try-on video according to claim 1, characterized in that: The target clothing features are respectively fused with the human body features in the spatial dimension to obtain spatial fusion features, including: Splicing the target clothing features with the human body features in the spatial dimension to obtain spatial splicing features corresponding to each frame; Performing self-attention calculation on the spatial splicing features in the spatial dimension to obtain spatial fusion features; The step of fusing the spatial fusion features with the human body matching clothing features in the time dimension to obtain the time fusion features includes: Splicing each of the spatial fusion features with each of the human body matching clothing features in the time dimension to obtain a time splicing feature; A self-attention calculation is performed on the time splicing feature in the time dimension to obtain a time fusion feature.

3. The method for generating a virtual try-on video according to claim 2, characterized in that: The step of fusing the spatial fusion features with the human body matching clothing features in the time dimension to obtain the time fusion features includes: Extracting global semantic features of the target clothing from the clothing picture; Combining the global semantic features with the spatial fusion features to obtain spatial fusion features of semantic matching; The semantic matching fusion features are fused with the human body matching clothing features in the time dimension to obtain a time fusion feature.

4. The method for generating a virtual try-on video according to claim 3, characterized in that: The combining the global semantic feature into the spatial fusion feature comprises: A cross-attention calculation is performed on the global semantic feature and the spatial fusion feature to combine the global semantic feature into the spatial fusion feature.

5. The method for generating a virtual try-on video according to claim 4, characterized in that: The step of extracting target clothing features corresponding to the target clothing from the clothing picture includes: The clothing image is encoded using a pre-trained clothing encoding model to obtain target clothing features corresponding to the target clothing.

6. The method for generating a virtual try-on video according to claim 5, characterized in that: The target clothing features are respectively fused with the human body features in the spatial dimension to obtain spatial fusion features, the global semantic features are combined with the spatial fusion features to obtain semantic matching spatial fusion features, and the semantic matching fusion features are fused with the human body matching clothing features in the temporal dimension to obtain temporal fusion features, including: Performing feature processing on the human body features corresponding to each frame through each feature processing layer of the pre-trained image generation model, obtaining processed human body features corresponding to each frame that are fused with the target clothing features, fused with the global semantic features, and the features of each frame are kept consistent in time series; The input of the i-th feature processing layer of the image generation model is obtained by the following steps: The output of the i-1th feature processing layer corresponding to each frame is fused with the target clothing feature in the spatial dimension to obtain the i-1th spatial fusion feature corresponding to each frame; Combining the global semantic feature with the i-1th spatial fusion feature to obtain the i-1th semantic matching spatial fusion feature corresponding to each frame; The i-1th semantic matching space fusion feature corresponding to each frame is fused with each of the human body matching clothing features in the time dimension to obtain the i-1th time fusion feature, and the i-1th time fusion feature is determined as the input of the i-th feature processing layer, i traverses 1 to N, N is the number of feature processing layers included in the image generation model, and the output of the 0th feature processing layer is the human body feature.

7. The method for generating a virtual try-on video according to claim 6, characterized in that: The clothing coding model includes a plurality of clothing coding layers; The step of extracting the global semantic features of the target clothing from the clothing picture includes: Extracting global semantic features of the target clothing from the clothing image through various visual extraction layers of a pre-trained visual model; The step of fusing the output of the i-1th feature processing layer corresponding to each frame with the target clothing feature in the spatial dimension to obtain the i-1th spatial fusion feature corresponding to each frame includes: The output of the i-1th feature processing layer corresponding to each frame is fused with the i-1th layer target clothing feature in the spatial dimension to obtain the i-1th spatial fusion feature corresponding to each frame, wherein the i-1th layer target clothing feature is the feature output by the i-1th layer clothing coding layer of the clothing coding model; The step of combining the global semantic feature with the i-1th spatial fusion feature to obtain the i-1th semantic matching spatial fusion feature corresponding to each frame includes: The i-1th spatial fusion feature is fused with the i-1th layer semantic feature to obtain the i-1th semantic matching spatial fusion feature corresponding to each frame, wherein the i-1th layer semantic feature is the feature output by the i-1th layer visual extraction layer of the visual model.

8. The method for generating a virtual try-on video according to claim 6, characterized in that: The step of extracting human features corresponding to each video frame from the human body video comprises: The clothing area in the video frame of the human body video is masked to obtain a clothing-agnostic video frame, and the clothing-agnostic video frame is determined as a human body feature corresponding to the video frame, wherein the hand area in the clothing-agnostic video frame is in an unmasked state.

9. The method for generating a virtual try-on video according to claim 8, characterized in that: The step of extracting human features corresponding to each video frame from the human body video further includes at least one of the following: Extracting a human body analysis result of each video frame from the human body video, and determining the human body analysis result as a human body feature corresponding to the video frame, wherein the human body analysis result is used to distinguish and mark various parts of the human body in the video frame; A mask corresponding to the clothing-agnostic video frame is determined, and the mask is determined as a human body feature corresponding to the video frame, wherein the mask is used to distinguish between a human body region and a masked clothing region in the clothing-agnostic video frame.

10. The method for generating a virtual try-on video according to claim 8, characterized in that: The step of masking the clothing region in the video frame of the human body video to obtain a clothing-agnostic video frame includes: Determine the human body key points, human body posture information and human body analysis results corresponding to each frame from the video frames of the human body video; Determine a clothing area in a video frame of the human body video according to the human body key points, the human body posture information, and the human body analysis result corresponding to the video frame; The clothing area in the video frame is masked to obtain a clothing-agnostic video frame.

11. The method for generating a virtual try-on video according to claim 9, characterized in that: The human body posture information is represented in the form of a human body posture graph; The process of performing feature processing on the human body features corresponding to each frame through each feature processing layer of the pre-trained image generation model includes: Extracting features from the human body posture graph using a pre-trained image feature extraction model to obtain human body posture features; A model input of a pre-trained image generation model is determined according to the human body posture features, the clothing agnostic video frame, and the mask, and the model is input into the image generation model to perform feature processing on the human body features corresponding to each frame through each feature processing layer of the image generation model.

12. The method for generating a virtual try-on video according to claim 11, characterized in that: The image generation model is a diffusion model; The step of determining a model input of a pre-trained image generation model according to the human body posture feature, the clothing agnostic video frame, and the mask comprises: Get random noise; A model input of a pre-trained image generation model is determined according to the random noise, the video frame of the human body video, the human body posture feature, the clothing agnostic video frame, and the mask, and the random noise is used to add noise to the video frame.

13. The method for generating a virtual try-on video according to any one of claims 1 to 12, characterized in that: Before extracting the target clothing features corresponding to the target clothing from the clothing picture and extracting the human body features corresponding to each video frame from the human body video, the method further includes: Dividing the human body video into a plurality of video segments, wherein there are overlapping video frames between two adjacent video segments; The virtual try-on video generation method generates a human try-on video frame for each video segment, and obtains a human try-on video frame corresponding to each video frame; For the overlapping video frame, the human body try-on video frame corresponding to the overlapping video frame is determined according to the previous human body try-on video frame and the subsequent human body try-on video frame corresponding to the overlapping video frame, the previous human body try-on video frame is the human body try-on video frame corresponding to the overlapping video frame calculated for the previous video segment to which the overlapping video frame belongs, and the subsequent human body try-on video frame is the human body try-on video frame corresponding to the overlapping video frame calculated for the subsequent video segment to which the overlapping video frame belongs.

14. The method for generating a virtual try-on video according to any one of claims 1 to 12, characterized in that: The step of adjusting the target clothing features based on the human body features to obtain human body matching clothing features that match the human body in each video frame includes: The target clothing features are respectively subjected to cross-attention calculation with each of the human body features to obtain each human body matching clothing feature that matches the human body in each video frame.

15. A model training method, characterized in that: The method comprises: Acquire training samples, wherein the training samples include a sample human body video, a sample clothing picture corresponding to a sample target clothing, and a sample fitting video of a person fitting the sample target clothing; Extracting sample target clothing features corresponding to the sample target clothing from the sample clothing picture, and extracting sample human body features corresponding to each sample video frame from the sample human body video; Based on each of the sample human body features, feature adjustment is performed on the sample target clothing features to obtain each sample human body matching clothing feature that matches the human body in each sample video frame; Performing feature processing on the sample human body features corresponding to each frame through each feature processing layer of the image generation model to be trained, obtaining output processed human body features corresponding to each frame that are integrated with the sample target clothing features and the features of each frame are consistent in time series, and determining each output human body fitting video frame fitting the sample target clothing according to the output processed human body features; Based on the consistency between the output human fitting video frame and each sample video frame of the sample fitting video, adjusting the parameters of the image generation model to be trained to obtain a trained image generation model; The input of the i-th feature processing layer of the image generation model to be trained is obtained by the following steps: The output of the i-1th feature processing layer corresponding to each sample frame is fused with the sample target clothing feature in the spatial dimension to obtain the i-1th spatial fusion feature of the sample corresponding to each sample frame; The i-1th spatial fusion feature of the sample corresponding to each sample frame is fused with the clothing feature matching each sample body in the time dimension to obtain the i-1th time fusion feature of the sample, and the i-1th time fusion feature of the sample is determined as the input of the i-th feature processing layer, i traverses 1 to N, N is the number of feature processing layers included in the image generation model to be trained, and the output of the 0th feature processing layer is the sample body feature.

16. The model training method according to claim 15, characterized in that: Before performing feature processing on the sample human features corresponding to each frame through each feature processing layer of the image generation model to be trained, the method further includes: Extracting sample global semantic features of the sample target clothing from the sample clothing picture; The method of performing feature processing on the sample human features corresponding to each frame through each feature processing layer of the image generation model to be trained, and obtaining output processed human features corresponding to each frame which are fused with the sample target clothing features and the features of each frame are kept consistent in time series, includes: Performing feature processing on the sample human body features corresponding to each frame through each feature processing layer of the image generation model to be trained, obtaining output processed human body features corresponding to each frame that are fused with the sample target clothing features and the sample global semantic features, and the features of each frame are kept consistent in time series; The step of fusing the sample i-1th spatial fusion feature corresponding to each sample frame with each sample body matching clothing feature in the time dimension to obtain the sample i-1th temporal fusion feature includes: The sample global semantic feature is combined with the sample i-1th spatial fusion feature to obtain the sample i-1th semantic matching spatial fusion feature corresponding to each sample frame; The i-1th semantic matching spatial fusion feature of the sample corresponding to each sample frame is fused with each human body matching clothing feature in the time dimension to obtain the i-1th time fusion feature of the sample.

17. The model training method according to claim 16, characterized in that: The step of extracting the sample target clothing features corresponding to the sample target clothing from the sample clothing picture includes: Performing clothing encoding on the sample clothing image by using a clothing encoding model to be trained to obtain sample target clothing features corresponding to the sample target clothing, wherein the clothing encoding model to be trained includes multiple clothing encoding layers to be trained; The step of fusing the output of the i-1th feature processing layer corresponding to each sample frame with the sample target clothing feature in the spatial dimension to obtain the i-1th spatial fusion feature of the sample corresponding to each sample frame includes: The output of the i-1th feature processing layer corresponding to each sample frame is fused with the i-1th target clothing feature of the sample in the spatial dimension to obtain the i-1th spatial fusion feature of the sample corresponding to each sample frame, wherein the i-1th target clothing feature of the sample is the feature output of the i-1th clothing coding layer to be trained of the clothing coding model to be trained; The model training method also includes: Based on the consistency between the output human fitting video frame and each sample video frame of the sample fitting video, the parameters of the clothing coding model to be trained are adjusted to obtain a trained clothing coding model.

18. The model training method according to claim 17, characterized in that: The step of extracting the sample global semantic features of the sample target clothing from the sample clothing picture includes: Extracting sample global semantic features of the sample target clothing from the sample clothing picture through each visual extraction layer of the visual model to be trained; The step of combining the sample global semantic features with the sample i-1th spatial fusion features to obtain the sample i-1th semantic matching spatial fusion features corresponding to each sample frame includes: The i-1th spatial fusion feature of the sample is fused with the i-1th layer semantic feature of the sample to obtain the i-1th semantic matching spatial fusion feature of the sample corresponding to each sample frame, wherein the i-1th layer semantic feature of the sample is the feature output by the i-1th layer visual extraction layer of the sample of the visual model to be trained; The model training method also includes: Based on the consistency between the output human fitting video frame and each sample video frame of the sample fitting video, the parameters of the visual model to be trained are adjusted to obtain a trained visual model.

19. The model training method according to claim 17, characterized in that: The image generation model to be trained is a diffusion model to be trained, the input of the diffusion model to be trained also includes preset noise, and the diffusion model to be trained is also used to output predicted noise; The model training method also includes: Based on the consistency between the predicted noise and the preset noise, parameters of the image generation model to be trained are adjusted to obtain a trained image generation model.

20. A model training method, characterized in that: The method comprises: Acquire training samples, wherein the training samples include a sample human body video, a sample clothing picture corresponding to a sample target clothing, and a sample fitting video of a person fitting the sample target clothing; Extracting sample target clothing features corresponding to the sample target clothing from the sample clothing picture, and extracting sample human body features corresponding to each sample video frame from the sample human body video; The sample human body features corresponding to each frame are processed by each feature processing layer of the image generation model to be trained to obtain the first output processed human body features integrated with the sample target clothing features, and the first output human body fitting video frames trying on the sample target clothing are determined according to the first output processed human body features; wherein the input of the i-th feature processing layer of the image generation model to be trained is obtained by the following steps: the output of the i-1th feature processing layer corresponding to each sample frame is integrated with the sample target clothing features in the spatial dimension to obtain the i-1th spatial fusion features of the samples corresponding to each sample frame, and the input of the i-th feature processing layer is determined according to the i-1th spatial fusion features, i traverses 1 to N, N is the number of feature processing layers included in the image generation model to be trained, and the output of the 0th feature processing layer is the sample human body features; Based on the consistency between the first output human fitting video frame and each sample video frame of the sample fitting video, adjusting the parameters of the image generation model to be trained to obtain the image generation model after the first stage of training; Based on each of the sample human body features, feature adjustment is performed on the sample target clothing features to obtain each sample human body matching clothing feature that matches the human body in each sample video frame; The sample human body features corresponding to each frame are processed by each feature processing layer of the image generation model after training in the first stage, and the second output processed human body features corresponding to each frame are obtained, which are integrated with the sample target clothing features and the features of each frame are consistent in the time series, and the second output human body fitting video frames trying on the sample target clothing are determined according to the second output processed human body features; wherein the input of the i-th feature processing layer of the image generation model after training in the first stage is obtained by the following steps: the output of the i-1th feature processing layer corresponding to each sample frame is integrated with the sample target clothing features in the spatial dimension to obtain the sample i-1th spatial fusion feature corresponding to each sample frame; the sample i-1th spatial fusion feature corresponding to each sample frame is integrated with each sample human body matching clothing feature in the time dimension to obtain the sample i-1th time fusion feature, and the sample i-1th time fusion feature is determined as the input of the i-th feature processing layer, i traverses 1 to N, N is the number of feature processing layers included in the image generation model to be trained, and the 0th sample spatial fusion feature is the sample human body feature; Based on the consistency between the second output human fitting video frame and each sample video frame of the sample fitting video, the parameters of the image generation model after training in the first stage are adjusted to obtain the trained image generation model.

21. An electronic device, characterized in that: include: A processor, a memory, and computer program instructions stored on the memory and executable on the processor; When the processor executes the computer program instructions, the method according to any one of claims 1 to 20 is implemented.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in any one of claims 1 to 20 above.

Images