Multi-modal fusion dynamic virtual fitting video generation method and device

Abstract

The invention provides a multi-modal fusion dynamic virtual fitting video generation method and device. The problems that static generation lacks dynamic authenticity and the multi-modal fusion effect is poor in the virtual fitting technology are effectively solved. The method comprises the steps of obtaining a to-be-synthesized clothing picture, a to-be-synthesized model picture and a target text, and generating a target picture based on the to-be-synthesized clothing picture and the to-be-synthesized model picture; extracting a plurality of clothing features from the target picture, and encoding the plurality of clothing features and the target text to obtain a plurality of clothing feature vectors and a target text semantic vector; mapping the various clothing feature vectors and the target text semantic vector to a submerged space to obtain a fused feature vector; and generating a corresponding video frame sequence based on the fusion feature vector layer by layer, so as to generate a dynamic clothing display video when the model executes the specified action with the to-be-synthesized clothing based on the video frame sequence.

Description

Technical Field

[0001] The present application relates to a field, specifically, to a method and device for generating dynamic virtual fitting videos with multimodal fusion. Background Art

[0002] In recent years, with the continuous development of e-commerce, virtual fitting has gradually entered people's field of vision. Although virtual fitting technology provides convenience for e-commerce shopping, its practical value is restricted by its technical bottlenecks. The specific problems are as follows: 1. Static generation lacks dynamic authenticity: Mainstream solutions are based on single-frame image synthesis (such as diffusion models), which can only display static clothing effects in fixed postures. They cannot simulate the natural swing, wrinkle changes and drape of clothing during movement, making it difficult for users to evaluate dynamic wearing effects (such as the flowing feel of a skirt or the elasticity of sportswear).

[0003] 2. Poor multimodal fusion effect: Cross-modal feature alignment is difficult, and the spatial matching accuracy between clothing images (2D / 3D) and human poses (OpenPose / SMPL, etc.) is insufficient, resulting in poor clothing fit (such as loose clothing slipping through the model or abnormal stretching during movement). The semantic correlation between text descriptions (such as "silk long skirt") and visual features (material, drape) is weak, and the generated results often deviate from expectations (such as describing "flowing" but rendering a stiff effect). Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method and device for generating dynamic virtual fitting videos with multimodal fusion. This method and device for generating dynamic virtual fitting videos with multimodal fusion effectively solve the problems of static generation lacking dynamic authenticity and poor multimodal fusion effect existing in virtual fitting technology.

[0005] In a first aspect, an embodiment of the present application provides a method for generating a dynamic virtual fitting video using multimodal fusion, the method comprising: Obtaining a garment image to be synthesized, a model image to be synthesized, and a target text, and generating a target image based on the garment image to be synthesized and the model image to be synthesized; the target image is a static image of the model to be synthesized wearing the garment to be synthesized; the target text includes a specified action; Extracting a plurality of clothing features from the target image, and encoding the plurality of clothing features and the target text to obtain a plurality of clothing feature vectors corresponding to the plurality of clothing features and a target text semantic vector corresponding to the target text; Mapping the multiple clothing feature vectors and the target text semantic vector to a latent space, and fusing the multiple clothing feature vectors and the target text semantic vector with temporal features in the latent space to obtain a fused feature vector; Based on the fused feature vectors, corresponding video frame sequences are hierarchically generated, so as to generate a clothing dynamic display video when a model wearing the clothing to be synthesized performs the specified action based on the video frame sequence.

[0006] In combination with the first aspect, an embodiment of the present application provides a first possible implementation of the first aspect, wherein the fusion of the multiple clothing feature vectors and the target text semantic vector by the temporal feature in the latent space to obtain a fused feature vector includes: Filtering the target text semantic vector and the multiple clothing feature vectors to obtain a key semantic vector and a local clothing feature vector corresponding to the key semantic vector; The key semantic vector and the local clothing feature vector are dynamically aligned to obtain a fused feature vector corresponding to the target text.

[0007] In combination with the first aspect, the embodiment of the present application provides a second possible implementation of the first aspect, wherein the dynamically aligning the key semantic vector with the local clothing feature vector includes: Splitting the key semantic vector to obtain a vector pair of target type, and assigning corresponding attention weights based on different types of vectors in the vector pair; A target clothing feature vector is selected from the local clothing feature vectors based on the attention weights to respond to the vector pair.

[0008] In combination with the first aspect, the embodiment of the present application provides a third possible implementation of the first aspect, wherein the step of hierarchically generating a corresponding video frame sequence based on the fused feature vector includes: Generate multiple low-resolution key frames based on the fused feature vector, and interpolate between the multiple key frames to generate intermediate frames, calling the target image as the first frame; Dynamic details between the key frame, the first frame, and the intermediate frames are enhanced to obtain the video frame sequence.

[0009] In combination with the first aspect, the embodiment of the present application provides a fourth possible implementation of the first aspect, wherein generating a clothing dynamic display video when a model wearing the garment to be synthesized performs the specified action based on the video frame sequence includes: Extracting the model's motion information from the video frame sequence to simulate the dynamic effect of the garment to be synthesized under a specified motion based on the motion information; The video frame sequence and the clothing dynamic effect are rendered and synthesized to obtain the clothing dynamic display video.

[0010] In combination with the first aspect, the embodiment of the present application provides a fifth possible implementation of the first aspect, wherein, in addition to mapping the multiple clothing feature vectors and the target text semantic vector to the latent space, the method further includes: Based on the clothing feature vector and the target text semantic vector, calling corresponding timestamp information and noise; The timestamp information and noise are encoded to obtain a corresponding embedding vector, so as to map the corresponding embedding vector to the latent space.

[0011] In combination with the first aspect, the embodiment of the present application provides a sixth possible implementation of the first aspect, wherein generating a target image based on the image of the garment to be synthesized and the image of the model to be synthesized includes: Extracting features from the garment image to be synthesized and the model image to be synthesized respectively, and inputting the features into a preset synthesis network; The target image is obtained by processing features in the clothing image to be synthesized and the model image to be synthesized based on the preset synthesis network.

[0012] In a second aspect, an embodiment of the present application provides a multimodal fusion dynamic virtual fitting video generation device, the device comprising: An acquisition module is used to acquire a picture of the garment to be synthesized, a picture of the model to be synthesized, and a target text, and generate a target picture based on the picture of the garment to be synthesized and the picture of the model to be synthesized; the target picture is a static picture of the model to be synthesized wearing the garment to be synthesized; the target text includes a specified action; an encoding module, configured to extract a plurality of clothing features from the target image, and encode the plurality of clothing features and the target text to obtain a plurality of clothing feature vectors corresponding to the plurality of clothing features and a target text semantic vector corresponding to the target text; A mapping module, configured to map the plurality of clothing feature vectors and the target text semantic vector to a latent space, and fuse the plurality of clothing feature vectors and the target text semantic vector within the latent space by using temporal features to obtain a fused feature vector; An execution module is used to hierarchically generate a corresponding video frame sequence based on the fused feature vector, so as to generate a clothing dynamic display video when a model wearing the clothing to be synthesized performs the specified action based on the video frame sequence.

[0013] In a third aspect, an embodiment of the present application provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, any one of the steps of the multimodal fusion dynamic virtual fitting video generation method is performed.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program executes any one of the steps of the multimodal fusion dynamic virtual fitting video generation method.

[0015] The embodiment of the present application provides a multimodal fusion dynamic virtual fitting video generation method, the method first obtains a picture of the clothing to be synthesized, a picture of the model to be synthesized and a target text, and generates a target picture based on the picture of the clothing to be synthesized and the picture of the model to be synthesized; the target picture is a static picture of the model to be synthesized wearing the clothing to be synthesized; the target text includes a specified action; secondly, multiple clothing features are extracted from the target picture, and the multiple clothing features and the target text are encoded to obtain multiple clothing feature vectors corresponding to the multiple clothing features and a target text semantic vector corresponding to the target text; then the multiple clothing feature vectors and the target text semantic vector are mapped to a latent space During the time, the multiple clothing feature vectors and the target text semantic vector are fused with the temporal features in the latent space to obtain a fused feature vector; finally, a corresponding video frame sequence is generated in layers based on the fused feature vector, so as to generate a clothing dynamic display video when the model wears the clothing to be synthesized and performs the specified action based on the video frame sequence, thereby ensuring the dynamic effect of the clothing dynamic display video in displaying the clothing to be synthesized, effectively solving the problems of lack of dynamic authenticity of static generation and poor multimodal fusion effect in virtual fitting technology, realizing a true simulation effect of dynamic clothing effect, ensuring the temporal continuity of video generation, and enhancing user interaction and generation controllability. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A schematic diagram of a process for generating a multimodal fusion dynamic virtual fitting video provided in an embodiment of the present application is shown; Figure 2 A flowchart of a multimodal fusion dynamic virtual fitting video generation method provided in an embodiment of the present application is shown; Figure 3 A schematic diagram of the multimodal fusion process provided by an embodiment of the present application is shown; Figure 4The following is a structural block diagram of a multimodal fusion dynamic virtual fitting video generation device provided in an embodiment of the present application; Figure 5 The figure shows a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0019] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0020] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0021] While current virtual fitting technology provides convenience for e-commerce shopping, its technical bottlenecks restrict its practical value. Specific issues include: static generation lacks dynamic authenticity, making it difficult for users to evaluate dynamic wearing effects; insufficient spatial matching accuracy between clothing images and human postures results in poor clothing fit; and weak semantic correlation between text descriptions and visual features, resulting in generated results that often deviate from expectations.

[0022] Based on this, the embodiments of the present application provide a method and device for generating a dynamic virtual fitting video with multimodal fusion, which will be described below through embodiments.

[0023] Example 1 To facilitate understanding of this embodiment, a multimodal fusion dynamic virtual fitting video generation method disclosed in the embodiment of this application is first introduced in detail. Figure 1 The flowchart of a method for generating a dynamic virtual fitting video using multimodal fusion is shown. The present application provides a method for generating a dynamic virtual fitting video using multimodal fusion, and the method includes: S101, obtaining a picture of a garment to be synthesized, a picture of a model to be synthesized, and a target text, and generating a target picture based on the picture of the garment to be synthesized and the picture of the model to be synthesized; the target picture is a static picture of the model to be synthesized wearing the garment to be synthesized; the target text includes a specified action; S102, extracting multiple clothing features from the target image, and encoding the multiple clothing features and the target text to obtain multiple clothing feature vectors corresponding to the multiple clothing features and a target text semantic vector corresponding to the target text; S103, mapping the multiple clothing feature vectors and the target text semantic vector to a latent space, and fusing the multiple clothing feature vectors and the target text semantic vector with temporal features in the latent space to obtain a fused feature vector; S104: Generate corresponding video frame sequences in layers based on the fused feature vectors, so as to generate a clothing dynamic display video when a model wearing the clothing to be synthesized performs the specified action based on the video frame sequence.

[0024] In step S101, Figure 2 As shown, the garment image to be synthesized, the model image to be synthesized, and the target text are all set according to user needs, wherein the garment image to be synthesized is a Garment Image, the model image to be synthesized is a Model Image, and the target text is a textual expression of the user needs. The target text includes a specified action, such as the target text can be "The woman is walking", in which case walking is the specified action. After obtaining the garment image to be synthesized and the model image to be synthesized, the garment image to be synthesized and the model image to be synthesized are aligned, that is, the garment image to be synthesized and the model image to be synthesized are preprocessed, and the garment image to be synthesized and the model image to be synthesized after preprocessing are used to generate a target image, which is a static image of the model to be synthesized wearing the garment to be synthesized; that is, the present application first generates the target image, and then uses the static target image as the first frame of the dynamic virtual fitting video, and then obtains the dynamic virtual fitting video based on the first frame.

[0025] In the specific implementation process of step S101, there is an embodiment in which the generating of the target image based on the garment image to be synthesized and the model image to be synthesized includes: S1011, extracting features from the garment image to be synthesized and the model image to be synthesized respectively, and inputting the features into a preset synthesis network; S1012: Process features in the clothing image to be synthesized and the model image to be synthesized based on the preset synthesis network to obtain the target image.

[0026] In steps S1011-S1012, after obtaining the clothing picture to be synthesized and the model picture to be synthesized, multiple features in the clothing picture to be synthesized and the model picture to be synthesized are respectively extracted, such as the feature in the clothing picture to be synthesized is clothing, including the texture, color, shape and light of the clothing, and the feature in the model picture to be synthesized is the model, including the posture, body shape, skin color and background information, and the features are input into a preset synthesis network, and the preset synthesis network can be OOTDiffusion, which is already trained. Then, after receiving the extracted multiple features in the clothing picture to be synthesized and the model picture to be synthesized, the preset synthesis network performs multi-scale fusion on the multiple features in the clothing picture to be synthesized and the model picture to be synthesized, such as splicing the clothing texture map and the model skin color map, and splicing the clothing feature vector and the model body shape code, and also weights the channels to highlight the key areas of the target picture to synthesize the target picture.

[0027] In step S102, a plurality of clothing features are extracted from the target image based on a preset feature extraction network, the clothing features including but not limited to texture, pattern, size and material attributes, and the plurality of clothing features and the target text are encoded to obtain a plurality of clothing feature vectors corresponding to the plurality of clothing features and a target text semantic vector corresponding to the target text, wherein the encoding of the plurality of clothing features is implemented based on an encoder of a VAE model, and the feature extraction network is also implemented based on a VAE model, and the encoding of the target text is implemented based on a CLIP text encoder, and the plurality of clothing feature vectors and the target text semantic vector are processed by the encoder, wherein the lengths of the plurality of clothing feature vectors and the target text semantic vector are fixed.

[0028] In step S103, after obtaining the multiple clothing feature vectors and the target text semantic vector, the multiple clothing feature vectors and the target text semantic vector are mapped to a latent space. In order to more accurately process the multiple clothing feature vectors and the target text semantic vector, the timestamp and noise in the encoder are respectively encoded to obtain a time embedding and a noise tensor, which are mapped together to the latent space, wherein the noise tensor comes from a standard Gaussian distribution. In the latent space, the multiple clothing feature vectors, the target text semantic vector, the time embedding, and the noise tensor are fused through multiple DiT Blocks (Diffusion Transformer modules) using temporal features to obtain a fused feature vector, thereby achieving processing for multimodal data, ensuring the effect of multimodal data fusion, and achieving cross-time step modeling of multimodal data. The multiple clothing feature vectors and the target text semantic vector are combined to guide the generation of each frame, ensuring action coherence and visual consistency. Based on the cross-modal cross-attention mechanism, the description of the target text (such as "running"), clothing attributes, and action sequence are aligned to generate a semantically consistent motion effect, ensuring that the dynamic effect matches user needs (such as "running" corresponds to a reasonable swing amplitude).

[0029] In the specific implementation process of step S103, there is an embodiment: Figure 3 As shown, in addition to mapping the multiple clothing feature vectors and the target text semantic vector to the latent space, the method also includes: S10311. Based on the clothing feature vector and the target text semantic vector, call corresponding timestamp information and noise; S10312: Encode the timestamp information and the noise to obtain a corresponding embedding vector, and map the corresponding embedding vector to the latent space.

[0030] In steps S10311-S10312, after obtaining the clothing feature vector and the target text semantic vector, the clothing feature vector and the target text semantic vector need to be processed. At this time, based on the clothing feature vector and the target text semantic vector, the corresponding timestamp information and noise are called, and the timestamp information and noise are encoded to obtain the corresponding embedding vector, wherein the timestamp information is first converted into a numerical feature and normalized before encoding. The timestamp information represents the modeling time sequence, controls the movement rhythm, enhances the consistency between frames, and avoids problems such as jumping and drifting. The corresponding embedding vectors are time embedding and noise tensor, and the time embedding and noise tensor are mapped to the latent space through the MLP neural network, thereby realizing the processing of multimodal data.

[0031] In the specific implementation process of step S103, there is another embodiment in which the fusing of the multiple clothing feature vectors and the target text semantic vector with the temporal features in the latent space to obtain a fused feature vector includes: S10321, screening the target text semantic vector and the multiple clothing feature vectors to obtain a key semantic vector and a local clothing feature vector corresponding to the key semantic vector; S10322. Dynamically align the key semantic vector and the local clothing feature vector to obtain a fused feature vector corresponding to the target text.

[0032] In steps S10321-S10322, a key semantic vector is screened from the target semantic vector based on NLP technology, and the clothing feature vector is decomposed into multiple local clothing feature vectors such as tops, skirts, etc. through a pre-trained local feature extractor. For example, the local clothing feature vector in "The woman is walking" is the swing amplitude. The key semantic vector is matched with the multiple clothing feature vectors through an attention mechanism, and a local clothing feature vector corresponding to the key semantic vector is screened based on the matching result. For example, "walking" corresponds to the swing amplitude of the skirt hem. The key semantic vector and the local clothing feature vector are dynamically aligned based on the cross attention mechanism to obtain a fused feature vector corresponding to the target text. That is, the weights of the local clothing feature vectors are adjusted based on the key semantic vector, and the generated feature vector is dynamically adjusted. That is, at each time step and spatial position of the generated video, it is dynamically determined which clothing areas (such as the corners of the clothes and the hem of the skirt) need to respond to the action semantics in the text, so that the action performance and the clothing response are naturally coordinated, thereby ensuring the effect of the fused feature vector.

[0033] In the specific implementation process of step S10322, there is an embodiment in which the dynamic alignment of the key semantic vector and the local clothing feature vector includes: S103221. Split the key semantic vector to obtain a vector pair of target type, and assign corresponding attention weights based on different types of vectors in the vector pair; S103222. Select a target clothing feature vector from the local clothing feature vectors based on the attention weight to respond to the vector pair.

[0034] In steps S103221-S103222, at each time step and spatial position, the key semantic vector is split to obtain a vector pair of target type, wherein the vector pair of target type is a verb type and a noun type, wherein the vector pair of verb type controls the motion type (such as "rotation" → periodic swing), and the noun feature locates the local clothing feature vector (such as "skirt" → lower body clothing area), and corresponding attention weights are assigned based on the different types of vectors in the vector pair; that is, the attention weights corresponding to the noun type vector and the verb type vector are different, and based on the attention weights, a target clothing feature vector is selected from the local clothing feature vector to respond to the vector pair, that is, the attention weight of the local clothing is queried based on the noun type vector, thereby generating an attention map of the local clothing, thereby controlling the dynamic effect of the local clothing, and thus dynamically determining which clothing areas need to respond to the action semantics in the text. The attention weights can also be adjusted according to the verb tense (such as progressive tense, perfect tense) or emotional intensity in the target text to control the intensity and duration of the dynamic effect.

[0035] In step S104, the fused feature vector is processed based on the deconvolution layer of the decoder to generate a corresponding video frame sequence in a hierarchical manner, that is, the video frame sequence is obtained by first generating multiple key frames and then interpolating to generate intermediate frames, thereby eliminating inter-frame jitter and maintaining motion continuity, and using median filtering or image restoration, which may help to eliminate residual artifacts. The video before and after interpolation can be compared to check the difference between frames, or the quality can be evaluated using indicators such as PSNR and SSIM, so as to generate a clothing dynamic display video when the model wears the clothing to be synthesized and performs the specified action based on the video frame sequence, thereby ensuring the video quality of the clothing dynamic display video and the dynamic display effect of the clothing to be synthesized, such as the natural swing of the clothing to be synthesized when walking, thereby forming a complete closed-loop process from static fitting to dynamic synthesis, and restoring the real wearing scene.

[0036] In the specific implementation process of step S104, there is an embodiment in which the step of hierarchically generating a corresponding video frame sequence based on the fused feature vector includes: S10411. Generate multiple low-resolution key frames based on the fused feature vector, and interpolate between the multiple key frames to generate intermediate frames, calling the target image as the first frame; S10412: Enhance the dynamic details between the key frame, the first frame, and the intermediate frames to obtain the video frame sequence.

[0037] In steps S10411-S10412, the decoder-based deconvolution layer maintains the sparsity of the time dimension based on time embedding, and obtains multiple key frames for the fused feature vector based on the global discriminator and the local discriminator. For example, only 8 key frames are generated for a 24fps video, and the intermediate frames between the key frames in the time dimension are obtained using a preset interpolation method, that is, it is first necessary to generate multiple candidate intermediate frames, and the multiple candidate intermediate frames are fused to obtain intermediate frames. The preset interpolation method can be an optical flow constrained interpolation method, and a preset timing model is used to enhance the dynamic details between the key frames, the first frame and the intermediate frames. The preset timing model is used to penalize incoherent motion trajectories and focus on areas with intense motion, and high resolution is used for complex motion areas. The static background is kept at low resolution, balancing computational efficiency and quality to obtain the video frame sequence. Key gestures can also be input according to user needs to generate key frames, and the decoder generates intermediate frames in real time to obtain the video frame sequence.

[0038] In the specific implementation process of step S104, there is another embodiment: generating a clothing dynamic display video when a model wearing the clothing to be synthesized performs the specified action based on the video frame sequence includes: S10421, extracting the model's motion information from the video frame sequence, and simulating the dynamic effect of the garment to be synthesized under a specified motion based on the motion information; S10422: Render and synthesize the video frame sequence and the clothing dynamic effect to obtain the clothing dynamic display video.

[0039] In steps S10421-S10422, a multi-view pose estimation network, such as OpenPose, is used to extract the model's motion information from the video frame sequence, and a parameterized model such as SMPL is used to reconstruct the extracted 2D joint point information into a 3D human body model, thereby realizing bone-cloth coupling constraints and enabling the clothing to deform naturally with the model's motion, so as to simulate the dynamic effect of the to-be-synthesized clothing under a specified motion based on the motion information; to avoid motion delay, the end-to-end processing delay is set to be less than a preset time threshold, thereby ensuring the continuity of the motion; realistic material properties are also added to the clothing based on the PBR material network; the lighting conditions of the video frames are analyzed, and the shadow and reflection effects are calculated to make the lighting of the synthesized video coordinated; neural networks such as motion blur compensation are applied to improve the visual quality of the rendered video, thereby obtaining the dynamic display video of the clothing.

[0040] Example 2 This application also provides a multi-modal fusion dynamic virtual fitting video generation device, such as Figure 4The block diagram of a multimodal fusion dynamic virtual fitting video generation device is shown. The functions implemented by this multimodal fusion dynamic virtual fitting video generation device correspond to the steps of executing a multimodal fusion dynamic virtual fitting video generation method on a terminal device. This device can be understood as a component of a server including a processor. The multimodal fusion dynamic virtual fitting video generation device described in this application includes: The acquisition module 401 is used to acquire a picture of the garment to be synthesized, a picture of the model to be synthesized, and a target text, and generate a target picture based on the picture of the garment to be synthesized and the picture of the model to be synthesized; the target picture is a static picture of the model to be synthesized wearing the garment to be synthesized; the target text includes a specified action; An encoding module 402 is configured to extract a plurality of clothing features from the target image, and encode the plurality of clothing features and the target text to obtain a plurality of clothing feature vectors corresponding to the plurality of clothing features and a target text semantic vector corresponding to the target text; A mapping module 403 is configured to map the plurality of clothing feature vectors and the target text semantic vector to a latent space, and fuse the plurality of clothing feature vectors and the target text semantic vector with temporal features in the latent space to obtain a fused feature vector; The execution module 404 is configured to hierarchically generate a corresponding video frame sequence based on the fused feature vector, so as to generate a clothing dynamic display video when a model wearing the garment to be synthesized performs the specified action based on the video frame sequence.

[0041] In a feasible implementation, the mapping module includes: A screening module, configured to screen the target text semantic vector and the multiple clothing feature vectors to obtain a key semantic vector and a local clothing feature vector corresponding to the key semantic vector; An alignment module is used to dynamically align the key semantic vector with the local clothing feature vector to obtain a fused feature vector corresponding to the target text.

[0042] In a feasible implementation manner, the mapping module further includes: A splitting module is used to split the key semantic vector to obtain a vector pair of the target type, and assign corresponding attention weights based on the different types of vectors in the vector pair; A selection module is configured to select a target clothing feature vector from the local clothing feature vectors based on the attention weights to respond to the vector pair.

[0043] In a feasible implementation, the execution module includes: An interpolation module, configured to generate a plurality of low-resolution key frames based on the fused feature vector, and to interpolate between the plurality of key frames to generate intermediate frames, with the target image being the first frame; The enhancement module is used to enhance the dynamic details between the key frame, the first frame and the intermediate frame to obtain the video frame sequence.

[0044] In a feasible implementation manner, the execution module further includes: A simulation module is used to extract the model's motion information from the video frame sequence, so as to simulate the dynamic effect of the garment to be synthesized under the specified motion based on the motion information; The rendering module is used to render and synthesize the video frame sequence and the clothing dynamic effect to obtain the clothing dynamic display video.

[0045] In a feasible implementation, the mapping module also includes: A calling module, configured to call corresponding timestamp information and noise based on the clothing feature vector and the target text semantic vector; An encoding module is configured to encode the timestamp information and noise to obtain a corresponding embedding vector, so as to map the corresponding embedding vector to the latent space.

[0046] In a feasible implementation, the acquisition module includes: Input modules and, for respectively extracting features from the garment image to be synthesized and the model image to be synthesized, and inputting the features into a preset synthesis network; A processing module is used to process features in the clothing image to be synthesized and the model image to be synthesized based on the preset synthesis network to obtain the target image.

[0047] Example 3 The present application also provides an electronic device, such as Figure 5 As shown, it includes: a processor 501, a memory 502 and a bus 503, the memory 502 stores machine-readable instructions executable by the processor 501, and when the electronic device is running, the processor 501 and the memory 502 communicate with each other through the bus 503, and when the machine-readable instructions are executed by the processor 501, any one of the steps of the multimodal fusion dynamic virtual fitting video generation method is executed.

[0048] Example 4 The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program executes the steps of any one of the methods for generating a multimodal fusion dynamic virtual fitting video.

[0049] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0050] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0051] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0052] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, platform server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0053] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A multimodal fusion dynamic virtual fitting video generation method, characterized in that: The method comprises: Obtaining a garment image to be synthesized, a model image to be synthesized, and a target text, and generating a target image based on the garment image to be synthesized and the model image to be synthesized; the target image is a static image of the model to be synthesized wearing the garment to be synthesized; the target text includes a specified action; Extracting a plurality of clothing features from the target image, and encoding the plurality of clothing features and the target text to obtain a plurality of clothing feature vectors corresponding to the plurality of clothing features and a target text semantic vector corresponding to the target text; Mapping the multiple clothing feature vectors and the target text semantic vector to a latent space, and fusing the multiple clothing feature vectors and the target text semantic vector with temporal features in the latent space to obtain a fused feature vector; Based on the fused feature vectors, corresponding video frame sequences are hierarchically generated, so as to generate a clothing dynamic display video when a model wearing the clothing to be synthesized performs the specified action based on the video frame sequence.

2. The method according to claim 1, characterized in that The fusing of the multiple clothing feature vectors and the target text semantic vector with the temporal features in the latent space to obtain a fused feature vector includes: Filtering the target text semantic vector and the multiple clothing feature vectors to obtain a key semantic vector and a local clothing feature vector corresponding to the key semantic vector; The key semantic vector and the local clothing feature vector are dynamically aligned to obtain a fused feature vector corresponding to the target text.

3. The method according to claim 2, characterized in that The dynamically aligning the key semantic vector with the local clothing feature vector includes: Splitting the key semantic vector to obtain a vector pair of target type, and assigning corresponding attention weights based on different types of vectors in the vector pair; A target clothing feature vector is selected from the local clothing feature vectors based on the attention weights to respond to the vector pair.

4. The method according to claim 1, wherein The step of hierarchically generating a corresponding video frame sequence based on the fused feature vector comprises: Generate multiple low-resolution key frames based on the fused feature vector, and interpolate between the multiple key frames to generate intermediate frames, calling the target image as the first frame; Dynamic details between the key frame, the first frame, and the intermediate frames are enhanced to obtain the video frame sequence.

5. The method according to claim 1, wherein The step of generating a clothing dynamic display video based on the video frame sequence when a model wearing the clothing to be synthesized performs the specified action comprises: Extracting the model's motion information from the video frame sequence to simulate the dynamic effect of the garment to be synthesized under a specified motion based on the motion information; The video frame sequence and the clothing dynamic effect are rendered and synthesized to obtain the clothing dynamic display video.

6. The method according to claim 1, characterized in that In addition to mapping the multiple clothing feature vectors and the target text semantic vector to the latent space, the method further includes: Based on the clothing feature vector and the target text semantic vector, calling corresponding timestamp information and noise; The timestamp information and noise are encoded to obtain a corresponding embedding vector, so as to map the corresponding embedding vector to the latent space.

7. The method according to claim 1, characterized in that The generating of the target image based on the garment image to be synthesized and the model image to be synthesized comprises: Extracting features from the garment image to be synthesized and the model image to be synthesized respectively, and inputting the features into a preset synthesis network; The target image is obtained by processing features in the clothing image to be synthesized and the model image to be synthesized based on the preset synthesis network.

8. A multimodal fusion dynamic virtual fitting video generation device, characterized in that: The device comprises: An acquisition module is used to acquire a picture of the garment to be synthesized, a picture of the model to be synthesized, and a target text, and generate a target picture based on the picture of the garment to be synthesized and the picture of the model to be synthesized; the target picture is a static picture of the model to be synthesized wearing the garment to be synthesized; the target text includes a specified action; an encoding module, configured to extract a plurality of clothing features from the target image, and encode the plurality of clothing features and the target text to obtain a plurality of clothing feature vectors corresponding to the plurality of clothing features and a target text semantic vector corresponding to the target text; A mapping module, configured to map the plurality of clothing feature vectors and the target text semantic vector to a latent space, and fuse the plurality of clothing feature vectors and the target text semantic vector within the latent space by using temporal features to obtain a fused feature vector; An execution module is used to hierarchically generate a corresponding video frame sequence based on the fused feature vector, so as to generate a clothing dynamic display video when a model wearing the clothing to be synthesized performs the specified action based on the video frame sequence.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, the steps of the method for generating a multimodal fusion dynamic virtual fitting video are performed as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the method for generating a multimodal fusion dynamic virtual fitting video according to any one of claims 1 to 7.

Images