Method, device and equipment for virtually fitting clothes, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence, and provides a virtual garment fitting method and device, equipment, a storage medium and a program product, so that the problems of garment dislocation and artifact of a fitting image under the conditions of complex postures and the like can be improved, and the coherence of a synthetic video is improved. The method comprises the following steps: obtaining a multi-scale feature map based on a current frame of human body picture and a clothing picture, optimizing an optical flow field and an attention map according to the feature maps from small scale to large scale, obtaining a target optical flow field and a target attention map, and obtaining a preliminary fitting result of the current frame of human body picture; according to the preliminary fitting result of the current frame of human body picture, the preliminary fitting result of the historical frame of human body picture and the optimized fitting result of the historical frame of human body picture, obtaining a current frame key feature map, a historical frame key feature map and a historical frame value feature map; and according to the similarity between the current frame key feature map and the historical frame key feature map, performing weighted summation on the historical frame value feature map to obtain an optimized fitting result of the current frame human body picture.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer equipment, storage medium and computer program product for virtual clothing try-on. Background Art

[0002] With the development of artificial intelligence technology, computer vision is increasingly entering people's lives. From automatic navigation to facial payment, from intelligent object recognition to one-click image cutting, these technical applications of computer vision are influencing and changing people's living habits in all aspects.

[0003] In the field of e-commerce, people can buy clothes without leaving home. The emergence of virtual clothing try-on technology enables people to try on clothes in virtual space when shopping online.

[0004] Some virtual try-on technologies can be based on videos, directly replacing clothes based on the video. Users can upload a video of themselves and pictures of the clothes they want to try on, and with the help of this technology, they can get a synthetic video of trying on the clothes.

[0005] Traditional video-based virtual clothing try-on technology, in the process of generating predicted posture conversion results, due to the lack of guidance information, the patterns, texts, etc. on the clothing may be misplaced and deformed to a large extent, resulting in the lack of realism in the final generated video. Summary of the invention

[0006] Based on this, it is necessary to provide a method, device, computer equipment, storage medium and computer program product for virtually trying on clothing in response to the above technical problems.

[0007] The present application provides a method for virtually trying on clothing, the method comprising:

[0008] Get multiple frames of human body images in the video;

[0009] According to the clothing pictures and each frame of the human body picture, a preliminary fitting result of each frame of the human body picture is obtained;

[0010] Optimize the preliminary fitting result of each frame of the human body image to obtain the optimized fitting result of each frame of the human body image;

[0011] Wherein, when obtaining the preliminary fitting result of the current frame human body picture, obtaining the current frame human body feature map and clothing feature map of multiple scales based on the current frame human body picture and clothing picture, optimizing the optical flow field and the attention map according to the current frame human body feature map and clothing feature map from small scale to large scale, obtaining the target optical flow field and the target attention map, using the target optical flow field and the target attention map as guidance information, processing the current frame human body picture and clothing picture, and obtaining the preliminary fitting result of the current frame human body picture;

[0012] Among them, when optimizing the preliminary try-on results of the current frame human body picture, the preliminary try-on results of the current frame human body picture, the preliminary try-on results of the historical frame human body picture and the optimized try-on results of the historical frame human body picture are obtained; the current frame key feature map is obtained according to the preliminary try-on results of the current frame human body picture; the historical frame key feature map is obtained according to the preliminary try-on results of the historical frame human body picture; the historical frame value feature map is obtained according to the optimized try-on results of the historical frame human body picture; according to the similarity between the current frame key feature map and the historical frame key feature map, the historical frame value feature map is weightedly summed to obtain the optimized try-on result of the current frame human body picture.

[0013] In one embodiment, according to the human feature map and clothing feature map of the current frame from small scale to large scale, the optical flow field and the attention map are optimized to obtain the target optical flow field and the target attention map, including:

[0014] According to the human body feature map of the current frame from small scale to large scale, the self-optical flow field and self-attention map corresponding to each scale are obtained, and the self-optical flow field and self-attention map corresponding to the last scale are used as the target self-optical flow field and target self-attention map;

[0015] Among them, when the self-optical flow field and self-attention map corresponding to the current scale are obtained, if the current scale is not the minimum scale, the current frame human feature map of the current scale is obtained and fused with the current frame human feature map of the previous scale to obtain the current frame human fusion feature map, according to the self-optical flow field and self-attention map corresponding to the previous scale, the current frame human fusion feature map is transformed to obtain the current frame human conversion feature map, and the self-optical flow field and self-attention map corresponding to the current scale are obtained according to the current frame human conversion feature map;

[0016] According to the clothing feature map from small scale to large scale, the cross optical flow field and cross attention map corresponding to each scale are obtained, and the cross optical flow field and cross attention map corresponding to the last scale are used as the target cross optical flow field and target cross attention map;

[0017] Among them, when the cross optical flow field and the cross attention map corresponding to the current scale are obtained, if the current scale is not the minimum scale, the clothing feature map of the current scale is obtained and fused with the clothing feature map of the previous scale to obtain a clothing fusion feature map, and the clothing fusion feature map is transformed according to the cross optical flow field and the cross attention map corresponding to the previous scale to obtain a clothing conversion feature map, and the cross optical flow field and the cross attention map corresponding to the current scale are obtained according to the clothing conversion feature map; the previous scale is smaller than the current scale.

[0018] In one embodiment, the method further comprises:

[0019] When the self-optical flow field and the self-attention map corresponding to the current scale are obtained, if the current scale is not the minimum scale, the self-optical flow field and the self-attention map corresponding to the current scale are obtained according to the human feature map of the current frame of the current scale.

[0020] In one embodiment, the method further comprises:

[0021] When the cross optical flow field and the cross attention map corresponding to the current scale are obtained, if the current scale is the minimum scale, the cross optical flow field and the cross attention map corresponding to the current scale are obtained according to the clothing feature map of the current scale.

[0022] In one embodiment, the target optical flow field and the target attention map are used as guidance information to process the current frame human body image and clothing image to obtain a preliminary fitting result of the current frame human body image, including:

[0023] Shallowly encode the human body image and clothing image of the current frame respectively to obtain a shallowly encoded human body feature map and a shallowly encoded clothing feature map of the current frame;

[0024] Perform deformation attention distortion according to the target self-optical flow field, the target self-attention map and the shallowly encoded human feature map of the current frame to obtain a target human clothing feature map of the current frame;

[0025] Performing deformation attention distortion according to the target cross optical flow field, the target cross attention map and the shallowly encoded clothing feature map to obtain a target clothing feature map;

[0026] Merge the target human clothing feature map and the target clothing feature map of the current frame to obtain a merged feature map of the current frame;

[0027] The merged feature map of the current frame is shallowly decoded to obtain a preliminary fitting result of the human body image of the current frame.

[0028] In one embodiment, according to the similarity between the current frame key feature map and the historical frame key feature map, the historical frame value feature map is weighted summed to obtain the optimized fitting result of the current frame human body picture, including:

[0029] According to the relative size of the similarity between the current frame key feature graph and each historical frame key feature graph, the relative size of the weight assigned to each historical frame value feature graph is determined to obtain the weight of each historical frame value feature graph; the greater the similarity, the greater the weight;

[0030] According to the weights of the feature maps of each historical frame value, the feature maps of each historical frame value are weighted and summed to obtain the optimized fitting result of the current frame human body image.

[0031] The present application provides a device for virtually trying on clothing, the device comprising:

[0032] The human body image acquisition module is used to obtain multiple frames of human body images in the video;

[0033] An initial fitting processing module is used to obtain a preliminary fitting result for each frame of the human body image based on the clothing image and each frame of the human body image;

[0034] An optimized fitting processing module is used to optimize the preliminary fitting result of each frame of the human body image to obtain an optimized fitting result of each frame of the human body image;

[0035] Wherein, when obtaining the preliminary fitting result of the current frame human body picture, the initial fitting processing module is further used to: obtain the current frame human body feature map and clothing feature map of multiple scales based on the current frame human body picture and clothing picture, optimize the optical flow field and the attention map according to the current frame human body feature map and clothing feature map from small scale to large scale, obtain the target optical flow field and the target attention map, use the target optical flow field and the target attention map as guidance information, process the current frame human body picture and the clothing picture, and obtain the preliminary fitting result of the current frame human body picture;

[0036] Among them, when optimizing the preliminary try-on results of the current frame human body picture, the optimized try-on processing module is also used to: obtain the preliminary try-on results of the current frame human body picture, the preliminary try-on results of the historical frame human body picture and the optimized try-on results of the historical frame human body picture; obtain the current frame key feature map according to the preliminary try-on results of the current frame human body picture; obtain the historical frame key feature map according to the preliminary try-on results of the historical frame human body picture; obtain the historical frame value feature map according to the optimized try-on results of the historical frame human body picture; according to the similarity between the current frame key feature map and the historical frame key feature map, perform weighted summation on the historical frame value feature map to obtain the optimized try-on result of the current frame human body picture.

[0037] The present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the above method.

[0038] The present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to execute the above method.

[0039] The present application provides a computer program product, on which a computer program is stored, and the computer program is executed by a processor to perform the above method.

[0040] The above-mentioned virtual clothing try-on method, device, computer equipment, storage medium and computer program product obtain multiple frames of human body pictures in the video; obtain preliminary try-on results of each frame of human body picture according to the clothing picture and each frame of human body picture; optimize the preliminary try-on results of each frame of human body picture to obtain the optimized try-on results of each frame of human body picture; when obtaining the preliminary try-on results of the current frame of human body picture, obtain the current frame human body feature map and clothing feature map of multiple scales based on the current frame of human body picture and clothing picture, optimize the optical flow field and attention map according to the current frame human body feature map and clothing feature map from small scale to large scale, obtain the target optical flow field and target attention map, use the target optical flow field and target attention map as guidance information, process the current frame of human body picture and clothing picture, and obtain the preliminary try-on results of the current frame of human body picture. The problems of clothing misalignment and obvious artifacts in the try-on images under complex postures and large deformations are improved; when optimizing the preliminary try-on results of the current frame human body image, the preliminary try-on results of the historical frame human body image and the optimized try-on results of the historical frame human body image are obtained; the current frame key feature map is obtained according to the preliminary try-on results of the current frame human body image; the historical frame key feature map is obtained according to the preliminary try-on results of the historical frame human body image; the historical frame value feature map is obtained according to the optimized try-on results of the historical frame human body image; according to the similarity between the current frame key feature map and the historical frame key feature map, the historical frame value feature map is weightedly summed to obtain the optimized try-on results of the current frame human body image, and the initial try-on results of the current frame are optimized in combination with the feature information of the historical frames to improve the coherence of the synthesized video. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A schematic diagram of a flow chart of a method for virtually trying on clothing in one embodiment;

[0043] Figure 2 A schematic diagram of a processing architecture of STMVN in one embodiment;

[0044] Figure 3 A schematic diagram of a processing architecture of DAFVN in one embodiment;

[0045] Figure 4 is a structural block diagram of a device for virtually trying on clothing in one embodiment;

[0046] Figure 5FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] The method for virtual clothing try-on provided in the present application can be executed by a computer device. The method includes: Figure 1 The steps shown are:

[0049] Step S101, obtaining multiple frames of human body images in the video.

[0050] Users can upload videos containing themselves, and the computer equipment can obtain multiple frames of human body images from the videos.

[0051] After obtaining multiple frames of human body images, they can be processed by the space-time memory video virtual try-on network constructed by this application to first obtain the preliminary try-on results of each frame of human body images, and then obtain the optimized try-on results of each frame of human body images; the optimized try-on results of each frame of human body images form a synthetic video. The full name of the space-time memory video virtual try-on network in English is: Space-Time Memory Video Virtual Try-On Network, abbreviated as STMVN in English. The model architecture diagram of STMVN is shown in the figure below. Figure 2 shown.

[0052] Step S102, obtaining a preliminary fitting result of each frame of the human body image according to the clothing image and each frame of the human body image.

[0053] Among them, when the preliminary fitting result of the current frame human body picture is obtained, the current frame human body feature map and clothing feature map of multiple scales are obtained based on the current frame human body picture and the clothing picture, and the optical flow field and the attention map are optimized from small scale to large scale of the current frame human body feature map and the clothing feature map to obtain the target optical flow field and the target attention map. The current frame human body picture and the clothing picture are processed with the target optical flow field and the target attention map as guidance information to obtain the preliminary fitting result of the current frame human body picture.

[0054] The clothing picture includes clothing that the user wants to try on, and the clothing can be called target clothing.

[0055] Taking the current frame as the i-th frame as an example, the clothing image and the i-th frame human image can be input into STMVN. STMVN includes a human key point detection network and a human parsing network.

[0056] By processing the i-th frame of human body image through the human body key point detection network, the i-th frame of human body key point information can be obtained; the i-th frame of human body key point information is converted into the i-th frame of human body posture map, and the i-th frame of human body posture map can be a 3-channel RGB human body posture map.

[0057] By processing the i-th frame of the human body image through the human body parsing network, the i-th frame of human body parsing information can be obtained; the i-th frame of human body parsing image is converted into the i-th frame of human body clothing mask image; specifically, the human body area that needs to be covered by clothing in the i-th frame of human body parsing image is determined, and the pixels of these human body areas are set to black pixels to obtain the image without human body clothing, that is, the i-th frame of human body clothing mask image. If the target clothing is a top, the human body area that needs to be covered by clothing includes at least the left arm, the right arm, and the part from the neck to the lower limbs.

[0058] Then, STMVN can splice the i-th frame human pose image and the i-th frame human clothing mask image, and the splicing result can be called the i-th frame human splicing image. STMVN also includes a try-on network based on deformable attention flows (DAFVN). STMVN can input the i-th frame human splicing image and clothing image into DAFVN to obtain the preliminary try-on result of the i-th frame human image to solve the problem of clothing misalignment and artifacts in complex postures and large deformations.

[0059] After the i-th frame human clothing mask image and the i-th frame human clothing mask image are spliced ​​to form the i-th frame human splicing image, they are sent to DAFVN. The i-th frame human splicing image and clothing image can guide the generation of optical flow field and attention map, and provide a reference for predicting the distortion degree of target clothing and pixel prediction of human torso.

[0060] DAFVN obtains the preliminary fitting result of the i-th frame of the human body splicing image and the clothing image. The specific process is as follows:

[0061] DAFVN can include two pyramid feature extractors, namely the source pyramid feature extractor and the reference pyramid feature extractor. By inputting the clothing image into the source pyramid feature extractor, clothing feature maps of multiple scales can be obtained. By inputting the i-th frame human body splicing image into the reference pyramid feature extractor, human body feature maps of multiple scales can be obtained. According to the i-th frame human body feature map and clothing feature map from small scale to large scale, the optical flow field and attention map are optimized to obtain the target optical flow field and target attention map. Then, the target optical flow field and target attention map are used as guidance information to process the i-th frame human body image and clothing image to obtain the preliminary fitting result of the i-th frame human body image.

[0062] In the above manner, a preliminary fitting result of each frame of the human body picture can be obtained, and the process proceeds to step S103.

[0063] Step S103, optimizing the preliminary fitting result of each frame of the human body image to obtain the optimized fitting result of each frame of the human body image.

[0064] Among them, when optimizing the preliminary try-on results of the current frame human body picture, the preliminary try-on results of the current frame human body picture, the preliminary try-on results of the historical frame human body picture and the optimized try-on results of the historical frame human body picture are obtained; the current frame key feature map is obtained according to the preliminary try-on results of the current frame human body picture; the historical frame key feature map is obtained according to the preliminary try-on results of the historical frame human body picture; the historical frame value feature map is obtained according to the optimized try-on results of the historical frame human body picture; according to the similarity between the current frame key feature map and the historical frame key feature map, the historical frame value feature map is weightedly summed to obtain the optimized try-on result of the current frame human body picture.

[0065] Taking the current frame as the i-th frame as an example, the preliminary fitting results of the i-th frame human body picture, the preliminary fitting results of several historical frame human body pictures before the i-th frame, and the optimized fitting results of several historical frame human body pictures before the i-th frame can be obtained.

[0066] STMVN can also include a Space-Time Correspondence Try-On Network (STCN) based on a memory model.

[0067] DAFVN performs very well in solving the misalignment and artifact problems in complex postures and large deformations. However, since DAFVN does not utilize the spatiotemporal information in the video, the video lacks coherence. Therefore, this application uses STCN to use the information of historical frames to correct the mispredicted pixels in the preliminary fitting results of the current frame, reduce the misalignment of the predicted clothing and human body between frames, and thus make the synthesized video more coherent. The STCN of this application can optimize the preliminary fitting results more concisely and effectively.

[0068] The STCN includes several key encoders, which can be the same. The preliminary trial results of the i-th frame human body picture are input into one of the key encoders, and the result obtained is called the i-th frame key feature map. The preliminary trial results of several historical frame human body pictures are respectively input into each key encoder to obtain each historical frame key feature map. Each historical frame key feature map can be stored in the memory.

[0069] STCN may also include several value encoders, which may be the same. The optimized fitting results of several historical frames of human body pictures are input into each value encoder respectively to obtain a value feature map of each historical frame. Each historical frame value feature map may be stored in a memory.

[0070] Next, STCN can calculate the similarity between the current frame key feature map and each historical frame key feature map, determine the weight of the corresponding historical frame value feature map based on the similarity, and then perform weighted summation of each historical frame value feature map according to the weight, and the weighted summation result is called the weighted value feature map of the i-th frame.

[0071] STCN may also include a decoder, and the weighted feature map of the i-th frame is input into the decoder, and the obtained result is used as the optimized fitting result of the i-th frame human body picture.

[0072] In the above-mentioned method for virtual clothing try-on, multiple frames of human body images in the video are obtained; based on the clothing images and each frame of human body images, a preliminary try-on result of each frame of human body image is obtained; the preliminary try-on result of each frame of human body image is optimized to obtain the optimized try-on result of each frame of human body image; when the preliminary try-on result of the current frame of human body image is obtained, a current frame human body feature map and a clothing feature map of multiple scales are obtained based on the current frame of human body image and the clothing image, and the optical flow field and the attention map are optimized according to the current frame of human body feature map and the clothing feature map from small scale to large scale to obtain the target optical flow field and the target attention map, and the current frame of human body image and the clothing image are processed with the target optical flow field and the target attention map as guidance information to obtain the preliminary try-on result of the current frame of human body image, so that complex postures and large deformations can be eliminated. In this case, the problems of clothing misalignment and obvious artifacts in the try-on images are improved; when optimizing the preliminary try-on results of the current frame human body image, the preliminary try-on results of the current frame human body image, the preliminary try-on results of the historical frame human body image and the optimized try-on results of the historical frame human body image are obtained; according to the preliminary try-on results of the current frame human body image, the current frame key feature map is obtained; according to the preliminary try-on results of the historical frame human body image, the historical frame key feature map is obtained; according to the optimized try-on results of the historical frame human body image, the historical frame value feature map is obtained; according to the similarity between the current frame key feature map and the historical frame key feature map, the historical frame value feature map is weightedly summed to obtain the optimized try-on results of the current frame human body image, and the initial try-on results of the current frame are optimized in combination with the feature information of the historical frames to improve the coherence of the synthesized video.

[0073] In one embodiment, according to the human feature map and clothing feature map of the current frame from small scale to large scale, the optical flow field and the attention map are optimized to obtain the target optical flow field and the target attention map, including:

[0074] According to the human body feature map of the current frame from small scale to large scale, the self-optical flow field and self-attention map corresponding to each scale are obtained, and the self-optical flow field and self-attention map corresponding to the last scale are used as the target self-optical flow field and target self-attention map;

[0075] Among them, when the self-optical flow field and self-attention map corresponding to the current scale are obtained, if the current scale is not the minimum scale, the current frame human feature map of the current scale is obtained and fused with the current frame human feature map of the previous scale to obtain the current frame human fusion feature map, according to the self-optical flow field and self-attention map corresponding to the previous scale, the current frame human fusion feature map is transformed to obtain the current frame human conversion feature map, and the self-optical flow field and self-attention map corresponding to the current scale are obtained according to the current frame human conversion feature map;

[0076] According to the clothing feature map from small scale to large scale, the cross optical flow field and cross attention map corresponding to each scale are obtained, and the cross optical flow field and cross attention map corresponding to the last scale are used as the target cross optical flow field and target cross attention map;

[0077] Among them, when the cross optical flow field and the cross attention map corresponding to the current scale are obtained, if the current scale is not the minimum scale, the clothing feature map of the current scale is obtained and fused with the clothing feature map of the previous scale to obtain a clothing fusion feature map, and the clothing fusion feature map is transformed according to the cross optical flow field and the cross attention map corresponding to the previous scale to obtain a clothing conversion feature map, and the cross optical flow field and the cross attention map corresponding to the current scale are obtained according to the clothing conversion feature map; the previous scale is smaller than the current scale.

[0078] Specifically, the method provided by the present application also includes:

[0079] When the self-optical flow field and the self-attention map corresponding to the current scale are obtained, if the current scale is not the minimum scale, the self-optical flow field and the self-attention map corresponding to the current scale are obtained according to the human feature map of the current frame of the current scale.

[0080] Specifically, the method provided by the present application also includes:

[0081] When the cross optical flow field and the cross attention map corresponding to the current scale are obtained, if the current scale is the minimum scale, the cross optical flow field and the cross attention map corresponding to the current scale are obtained according to the clothing feature map of the current scale.

[0082] Combination Figure 3 , an example is given in which the current frame is the i-th frame and the number of scales is 3.

[0083] Figure 3 Schematic diagram of DAFVN processing framework; see Figure 3 ,DAFVN can sample multiple streams at the same time, extract feature-level information and pixel-level information of different semantic regions through the deformation attention mechanism, and can distort clothing and synthesize human bodies at the same time. The preliminary fitting results generated by DAFVN are of high quality, which is conducive to the optimization processing in the next stage.

[0084] DAFVN includes a reference pyramid feature extractor, which can input the i-th frame human posture map and the i-th frame human clothing mask map into the reference pyramid feature extractor to obtain a large-scale i-th frame human feature map, a medium-scale i-th frame human feature map and a small-scale i-th frame human feature map.

[0085] DAFVN includes a deformable attention flows network (DAFN). The i-th frame human feature map of a small scale can be input into DAFN to obtain the self-optical flow field corresponding to the small scale. and self-attention map Next, the middle-scale i-th frame human feature map is fused with the small-scale i-th frame human feature map to obtain a middle-scale-small-scale i-th frame human fusion feature map.

[0086] DAFVN also includes a deformable attention warp (DAWarp). It can fuse the medium-scale and small-scale i-th frame human body feature map, self-optical flow field and self-attention map Input into DAWarp to generate the optical flow field in DAWarp and self-attention map , convert the mesoscale-small-scale i-th frame human fusion feature map, and the result is called the mesoscale-small-scale i-th frame human body conversion feature map.

[0087] Then the middle-scale-small-scale i-th frame human body conversion feature map is input into DAFN to obtain the self-optical flow field corresponding to the middle scale and self-attention map Next, the large-scale i-th frame human feature map is fused with the medium-scale i-th frame human feature map to obtain a large-scale-medium-scale i-th frame human fusion feature map.

[0088] The i-th frame of the human body is fused with the large-scale and medium-scale feature maps and the self-optical flow field. and self-attention map Input into DAWarp to generate the optical flow field in DAWarp and self-attention map , convert the large-scale-medium-scale i-th frame human fusion feature map, and the result is called the large-scale-medium-scale i-th frame human body conversion feature map.

[0089] Then the large-scale-medium-scale i-th frame human body conversion feature map is input into DAFN to obtain the self-optical flow field corresponding to the medium scale and self-attention map . From optical flow field Optical flow field belonging to one of the target optical flow fields; self-attention map The attention map of one of the target attention maps.

[0090] DAFVN also includes a source pyramid feature extractor, into which clothing images can be input to obtain large-scale clothing feature maps, medium-scale clothing feature maps, and small-scale clothing feature maps.

[0091] The small-scale clothing feature map can be input into DAFN to obtain the cross optical flow field corresponding to the small scale. and cross attention map Then, the medium-scale clothing feature map is fused with the small-scale clothing feature map to obtain a medium-scale-small-scale clothing fusion feature map.

[0092] The medium-scale and small-scale clothing fusion feature maps and cross optical flow fields can be and cross attention map Input into DAWarp to generate the cross optical flow field in DAWarp and cross attention map , converting the mesoscale-small-scale clothing fusion feature map, the result is called the mesoscale-small-scale clothing conversion feature map.

[0093] Then the mesoscale-smallscale clothing conversion feature map is input into DAFN to obtain the cross optical flow field corresponding to the mesoscale and cross attention map Then, the large-scale clothing feature map is fused with the medium-scale clothing feature map to obtain a large-scale-medium-scale clothing fusion feature map.

[0094] Combine large-scale and medium-scale clothing feature maps and cross optical flow fields and cross attention map Input into DAWarp to generate the cross optical flow field in DAWarp and cross attention map , converting the large-scale-medium-scale clothing fusion feature map, the result is called the large-scale-medium-scale clothing conversion feature map.

[0095] Then the large-scale-medium-scale clothing conversion feature map is input into DAFN to obtain the cross optical flow field corresponding to the medium scale. and cross attention map Cross optical flow Optical flow field belonging to one of the target optical flow fields; Cross attention map The attention map that is one of the target attention maps.

[0096] In one embodiment, the target optical flow field and the target attention map are used as guidance information to process the current frame human body image and clothing image to obtain a preliminary fitting result of the current frame human body image, including:

[0097] Shallowly encode the human body image and clothing image of the current frame respectively to obtain a shallowly encoded human body feature map and a shallowly encoded clothing feature map of the current frame; perform deformation attention distortion according to the target self-optical flow field, the target self-attention map and the shallowly encoded human body feature map of the current frame to obtain the target human body clothing feature map of the current frame; perform deformation attention distortion according to the target cross-optical flow field, the target cross-attention map and the shallowly encoded clothing feature map to obtain the target clothing feature map; merge the target human body clothing feature map of the current frame and the target clothing feature map to obtain the merged feature map of the current frame; shallowly decode the merged feature map of the current frame to obtain the preliminary fitting result of the human body image of the current frame.

[0098] Taking the current frame as the i-th frame, the target optical flow field includes the self optical flow field and cross optical flow , and the target attention map includes the self-attention map and cross attention map For example.

[0099] DAFVN consists of a shallow encoder and a shallow decoder.

[0100] Input the human clothing mask image of the i-th frame into the shallow encoder to obtain the i-th shallow encoded human feature map; the i-th shallow encoded human feature map and the self-optical flow field and self-attention map Input DAWarp for deformation attention warping, and the result is called the target human clothing feature map of the i-th frame.

[0101] Input the clothing image into the shallow encoder to obtain the shallow encoded clothing feature map. and cross attention map Input DAWarp for deformation attention warping, and the result is called target clothing feature map.

[0102] The target human clothing feature map of the i-th frame and the target clothing feature map are merged to obtain a merged feature map of the i-th frame; the merged feature map of the i-th frame is input into a shallow decoder, and the result obtained is used as a preliminary fitting result of the i-th frame human body image.

[0103] In one embodiment, according to the similarity between the current frame key feature map and the historical frame key feature map, the historical frame value feature map is weighted summed to obtain the optimized fitting result of the current frame human body picture, including:

[0104] According to the relative size of the similarity between the current frame key feature map and each historical frame key feature map, the relative size of the weight assigned to each historical frame value feature map is determined to obtain the weight of each historical frame value feature map; the greater the similarity, the greater the weight; according to the weight of each historical frame value feature map, the weighted sum of each historical frame value feature map is performed to obtain the optimized fitting result of the current frame human body picture.

[0105] Take the current frame as the i-th frame as an example.

[0106] STCN includes several identical key encoders; STCN may also include several identical value encoders; STCN may also include a decoder.

[0107] The preliminary trial result of the i-th frame human body picture is input into one of the key encoders, and the result is called the i-th frame key feature map; the dimension size of the i-th frame key feature map is ; is the number of channels of the key feature map of the i-th frame, and are the height and width of the key feature map of the i-th frame respectively. The preliminary test results of several historical frames of human body pictures are input into each key encoder respectively to obtain the key feature map of each historical frame; the dimension size of each historical frame key feature map is ; is the number of channels of the historical frame key feature map, and are the height and width of the key feature map of the historical frame respectively. The optimized fitting results of several historical frame human body pictures are input into each value encoder respectively to obtain the value feature map of each historical frame; the dimension size of each historical frame value feature map is ; is the number of channels of the historical frame value feature map, and are the height and width of the historical frame key feature map respectively.

[0108] STCN can calculate the similarity between the key feature graph of the i-th frame and each key feature graph of the historical frame, obtain several similarities, and determine the relative size of the weight assigned to the corresponding historical frame value feature graph according to the relative size of the several similarities. Exemplarily, if the similarity between the key feature graph of the i-th frame and the key feature graph of the i-1th frame is greater than the similarity between the key feature graph of the i-th frame and the key feature graph of the i-2th frame, then the weight assigned to the i-1th frame value feature graph is greater than the weight assigned to the i-2th frame value feature graph. Thus, the weights of each historical frame value feature graph can be obtained, and the weighted summation of each historical frame value feature graph is performed according to the weight, and the weighted summation result is called the weighted value feature graph of the i-th frame. The weighted value feature graph of the i-th frame is input into the decoder, and the result obtained is used as the optimized fitting result of the i-th frame human body picture.

[0109] In order to construct the above STMVN, the training can be divided into three stages. The first stage is to train DAFVN, the second stage is to train the initial STCN in a static way, and the third stage is to train the final STCN in a dynamic way. A training set can be constructed, which includes 661 training videos and a total of 159,170 frames.

[0110] The following introduces the training sets used in each stage of training.

[0111] Specifically, the first stage of training aims to obtain a DAFVN that has the ability to predict preliminary fitting results. For example, all images in the training set can be used as the training set for the first stage of training, batch_size is set to 4, the number of training epochs is 20, the initial learning rate is set to 5×10^(-5), and is reduced to 0.2 times the original every 3 epochs, and the optimizer can use the Adam optimizer.

[0112] The second stage of static training aims to obtain an initial STCN with basic image generation capabilities. For example, 661 training videos in the training set can be obtained, and 30 frames of each training video are sampled to obtain 19,830 pictures. At the beginning of each epoch, the sampling is re-performed. For the 19,830 pictures generated by sampling, each picture is copied five times, thereby obtaining 19,830 five-frame videos, which are used as training sets for static training. The batch_size of the second stage training is set to 1, the epoch is 90, and the initial learning rate is 2×10^(-4), which is reduced to 0.2 times the original every 30 epochs.

[0113] The dynamic training in the third stage aims to obtain the final STCN that can generate realistic dressing videos. For example, 661 training videos in the training set can be obtained, and 5 frames are sampled for each training video. The interval between adjacent sampled frames in the training video does not exceed k frames, so that each epoch has 661 five-frame videos. The sampling interval k changes with the increase of the number of training epochs. The initial value is set to 5, and it increases by 5 every 600 epochs. When it reaches the last 600 epochs, it is reset to 5. At the beginning of each epoch, the 661 training videos in the training set are resampled, so that the 661 five-frame videos trained in each epoch can be regenerated and will not be repeated with the previous ones. The batch_size of the third stage training is 1, the epoch is 3000, and the initial learning rate is 2×10^(-4), which is reduced to 0.2 times the original every 1000 epochs.

[0114] The following introduces the loss functions used in each stage of training.

[0115] In the first stage, when training DAFVN, the loss function used can be obtained by weighting three loss functions, namely L1 loss function, perceptual loss function and style loss function.

[0116] The L1 loss function is obtained by calculating the L1 distance between the pre-test wear image and the real test wear image. The L1 loss function helps to optimize the details of the generated image. The calculation formula is shown in formula (1). represents the L1 distance, Represents the pre-test penetration diagram, This is a real try-on picture.

[0117] (1)

[0118] The perceptual loss function is obtained by calculating the L1 distance of 5 feature maps of different scales extracted from the pre-test wear image and the real wear image. The VGG-19 network can be used to extract 5 feature maps of different scales. The perceptual loss function helps to improve the structural authenticity of the generated image. Its calculation formula is shown in formula (2).

[0119] Represents the j-th scale feature map extracted by the VGG-19 network.

[0120] (2)

[0121] The style loss function calculates the L1 distance between the Gram feature matrices of feature maps of different scales extracted from the pre-test wear image and the real wear image to make the style of the pre-test wear image and the real wear image consistent. The Gram matrix can capture the texture and color information of the image, so the use of the style loss function can make the generated image more realistic and natural. Specifically, the calculation formula of the style loss function is shown in formula (3). Represents the Gram matrix of the j-th scale feature map.

[0122] (3)

[0123] The loss function used when training DAFVN is shown in formula (4). , and They are the weights of the L1 loss function, perceptual loss function, and style loss function respectively.

[0124] (4)

[0125] During the second and third phases of training, the loss function used aims to correct the pixels that were predicted incorrectly and improve the coherence of the synthesized video. Therefore, this application introduces the flow consistency loss function , to estimate the coherence loss between frames. The optical flow field can be divided into forward optical flow field and backward optical flow field , the forward optical flow field represents the optical flow from the previous frame to the next frame, and the backward optical flow field represents the optical flow from the next frame to the previous frame. It can be converted into and Alignment, represents the distortion operation, which is completed by bilinear interpolation. Similarly, we can also use (6) to calculate .

[0126] (5)

[0127] (6)

[0128] If the frames are coherent, then and are equal in magnitude and opposite in direction. and Also equal in magnitude and opposite in direction, then as shown in formula (7), the flow consistency loss function can be defined as:

[0129] (7)

[0130] in Indicates whether pixel p belongs to the human body area that needs to be covered by clothing. If so, The value of is 1, otherwise it is 0. In order to save the amount of calculation, the flow consistency loss function of this application can only calculate the human body area that needs to be covered by clothing. N represents the total number of pixels in the human body area that needs to be covered by clothing. If the coherence of the frame is better, then The closer the value is to 0.

[0131] In addition to the flow consistency loss function, the loss functions of the second and third stages also evaluate the quality of the generated video image itself, and the L1 loss function of formula (1) is applied. , the perceptual loss function of formula (2) , and the L2 loss function shown in (8) .

[0132] (8)

[0133] In order to improve the authenticity of the synthesized video, this application also uses a multi-scale discriminator based on a generative adversarial network to calculate the generative adversarial loss function The multi-scale discriminator of this application uses three discriminators with the same network structure but different discriminant scales, which are denoted as D1, D2, and D3. D1 accepts generated images of the original scale, D2 accepts generated images downsampled to 1 / 2 scale, and D3 accepts generated images downsampled to 1 / 4 scale. D3 has the largest receptive field and can perceive the structure of the generated image and improve the global consistency of the synthesis. D1 has the finest scale and can guide the generator to generate finer details. The calculation formula of is shown in formula (9). Indicates real video, Preliminary virtual try-on video representing the input to STCN.

[0134] (9)

[0135] The loss function used in the second and third stage training is shown in (10). , , , and is the weight.

[0136] (10)

[0137] After the three stages of training are completed, the DAFVN obtained from the first stage of training and the final STCN obtained from the third stage of training can be obtained to form DAFVN for reasoning scenarios.

[0138] In the reasoning scenario, when the virtual try-on video of clothing is generated by using the DAFVN provided by this application, the text and pattern of the clothing can be well preserved. This is because the DAFVN uses the extracted multi-level feature map to perform cascade update optimization on the two optical flow fields and attention maps, which can not only retain the high-level structural information of the clothing, but also preserve the low-level detail information. The solution provided by this application can well preserve the texture and material of the clothing. On the one hand, it is because DAFVN retains the low-level detail information, and on the other hand, because STCN has the ability to correct the wrongly predicted pixels, it can restore the texture and material characteristics of the clothing. The quality of the virtual try-on video of the clothing generated by the solution provided by this application is greatly improved compared with the traditional method.

[0139] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0140] Based on the same inventive concept, the embodiment of the present application also provides a virtual clothing try-on device for implementing the virtual clothing try-on method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more virtual clothing try-on device embodiments provided below can refer to the limitations of the virtual clothing try-on method above, and will not be repeated here.

[0141] In one embodiment, Figure 4 As shown, a device for virtually trying on clothing is provided, comprising:

[0142] A human body image acquisition module 401 is used to acquire multiple frames of human body images in a video;

[0143] An initial fitting processing module 402 is used to obtain a preliminary fitting result for each frame of the human body image based on the clothing image and each frame of the human body image;

[0144] The optimized fitting processing module 403 is used to optimize the preliminary fitting result of each frame of the human body image to obtain the optimized fitting result of each frame of the human body image;

[0145] Wherein, when obtaining the preliminary fitting result of the current frame human body picture, the initial fitting processing module 402 is further used to: obtain the current frame human body feature map and clothing feature map of multiple scales based on the current frame human body picture and clothing picture, optimize the optical flow field and the attention map according to the current frame human body feature map and clothing feature map from small scale to large scale, obtain the target optical flow field and the target attention map, use the target optical flow field and the target attention map as guidance information, process the current frame human body picture and the clothing picture, and obtain the preliminary fitting result of the current frame human body picture;

[0146] Among them, when optimizing the preliminary try-on results of the current frame human body picture, the optimized try-on processing module 403 is also used to: obtain the preliminary try-on results of the current frame human body picture, the preliminary try-on results of the historical frame human body picture and the optimized try-on results of the historical frame human body picture; obtain the current frame key feature map according to the preliminary try-on results of the current frame human body picture; obtain the historical frame key feature map according to the preliminary try-on results of the historical frame human body picture; obtain the historical frame value feature map according to the optimized try-on results of the historical frame human body picture; according to the similarity between the current frame key feature map and the historical frame key feature map, perform weighted summation on the historical frame value feature map to obtain the optimized try-on result of the current frame human body picture.

[0147] In one embodiment, the initial fitting processing module 402 is further used to:

[0148] According to the human body feature map of the current frame from small scale to large scale, the self-optical flow field and self-attention map corresponding to each scale are obtained, and the self-optical flow field and self-attention map corresponding to the last scale are used as the target self-optical flow field and target self-attention map; according to the clothing feature map from small scale to large scale, the cross optical flow field and cross attention map corresponding to each scale are obtained, and the cross optical flow field and cross attention map corresponding to the last scale are used as the target cross optical flow field and target cross attention map;

[0149] Among them, when the self-optical flow field and self-attention map corresponding to the current scale are obtained, if the current scale is not the minimum scale, the current frame human feature map of the current scale is obtained and fused with the current frame human feature map of the previous scale to obtain the current frame human fusion feature map, according to the self-optical flow field and self-attention map corresponding to the previous scale, the current frame human fusion feature map is transformed to obtain the current frame human conversion feature map, and the self-optical flow field and self-attention map corresponding to the current scale are obtained according to the current frame human conversion feature map;

[0150] Among them, when the cross optical flow field and the cross attention map corresponding to the current scale are obtained, if the current scale is not the minimum scale, the clothing feature map of the current scale is obtained and fused with the clothing feature map of the previous scale to obtain a clothing fusion feature map, and the clothing fusion feature map is transformed according to the cross optical flow field and the cross attention map corresponding to the previous scale to obtain a clothing conversion feature map, and the cross optical flow field and the cross attention map corresponding to the current scale are obtained according to the clothing conversion feature map; the previous scale is smaller than the current scale.

[0151] In one embodiment, the initial fitting processing module 402 is further used to:

[0152] When the self-optical flow field and the self-attention map corresponding to the current scale are obtained, if the current scale is not the minimum scale, the self-optical flow field and the self-attention map corresponding to the current scale are obtained according to the human feature map of the current frame of the current scale.

[0153] In one embodiment, the initial fitting processing module 402 is further used to:

[0154] When the cross optical flow field and the cross attention map corresponding to the current scale are obtained, if the current scale is the minimum scale, the cross optical flow field and the cross attention map corresponding to the current scale are obtained according to the clothing feature map of the current scale.

[0155] In one embodiment, the initial fitting processing module 402 is further used to:

[0156] Shallowly encode the human body image and clothing image of the current frame respectively to obtain a shallowly encoded human body feature map and a shallowly encoded clothing feature map of the current frame; perform deformation attention distortion according to the target self-optical flow field, the target self-attention map and the shallowly encoded human body feature map of the current frame to obtain a target human body clothing feature map of the current frame; perform deformation attention distortion according to the target cross-optical flow field, the target cross-attention map and the shallowly encoded clothing feature map to obtain a target clothing feature map; merge the target human body clothing feature map of the current frame and the target clothing feature map to obtain a merged feature map of the current frame; shallowly decode the merged feature map of the current frame to obtain a preliminary fitting result of the human body image of the current frame.

[0157] In one embodiment, the optimized fitting processing module 403 is further used to:

[0158] According to the relative size of the similarity between the current frame key feature map and each historical frame key feature map, the relative size of the weight assigned to each historical frame value feature map is determined to obtain the weight of each historical frame value feature map; the greater the similarity, the greater the weight; according to the weight of each historical frame value feature map, the weighted sum of each historical frame value feature map is performed to obtain the optimized fitting result of the current frame human body picture.

[0159] Each module in the above-mentioned virtual clothing try-on device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0160] In an exemplary embodiment, a computer device is provided, the internal structure of which can be shown as follows: Figure 5As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the above method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for virtually trying on clothing is implemented.

[0161] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0162] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned various method embodiments when executing the computer program.

[0163] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0164] In one embodiment, a computer program product is provided, on which a computer program is stored, and the computer program is used by a processor to execute the steps in the above-mentioned various method embodiments.

[0165] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0166] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0167] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0168] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for virtually trying on clothing, characterized in that: The method comprises: Get multiple frames of human body images in the video; According to the clothing pictures and each frame of the human body picture, a preliminary fitting result of each frame of the human body picture is obtained; Optimize the preliminary fitting result of each frame of the human body image to obtain the optimized fitting result of each frame of the human body image; Wherein, when obtaining the preliminary fitting result of the current frame human body picture, obtaining the current frame human body feature map and clothing feature map of multiple scales based on the current frame human body picture and clothing picture, optimizing the optical flow field and the attention map according to the current frame human body feature map and clothing feature map from small scale to large scale, obtaining the target optical flow field and the target attention map, using the target optical flow field and the target attention map as guidance information, processing the current frame human body picture and clothing picture, and obtaining the preliminary fitting result of the current frame human body picture; Among them, when optimizing the preliminary try-on results of the current frame human body picture, the preliminary try-on results of the current frame human body picture, the preliminary try-on results of the historical frame human body picture and the optimized try-on results of the historical frame human body picture are obtained; the current frame key feature map is obtained according to the preliminary try-on results of the current frame human body picture; the historical frame key feature map is obtained according to the preliminary try-on results of the historical frame human body picture; the historical frame value feature map is obtained according to the optimized try-on results of the historical frame human body picture; according to the similarity between the current frame key feature map and the historical frame key feature map, the historical frame value feature map is weightedly summed to obtain the optimized try-on result of the current frame human body picture.

2. The method according to claim 1, characterized in that According to the human feature map and clothing feature map of the current frame from small scale to large scale, the optical flow field and attention map are optimized to obtain the target optical flow field and target attention map, including: According to the human body feature map of the current frame from small scale to large scale, the self-optical flow field and self-attention map corresponding to each scale are obtained, and the self-optical flow field and self-attention map corresponding to the last scale are used as the target self-optical flow field and target self-attention map; Among them, when the self-optical flow field and self-attention map corresponding to the current scale are obtained, if the current scale is not the minimum scale, the current frame human feature map of the current scale is obtained and fused with the current frame human feature map of the previous scale to obtain the current frame human fusion feature map, according to the self-optical flow field and self-attention map corresponding to the previous scale, the current frame human fusion feature map is transformed to obtain the current frame human conversion feature map, and the self-optical flow field and self-attention map corresponding to the current scale are obtained according to the current frame human conversion feature map; According to the clothing feature map from small scale to large scale, the cross optical flow field and cross attention map corresponding to each scale are obtained, and the cross optical flow field and cross attention map corresponding to the last scale are used as the target cross optical flow field and target cross attention map; Among them, when the cross optical flow field and the cross attention map corresponding to the current scale are obtained, if the current scale is not the minimum scale, the clothing feature map of the current scale is obtained and fused with the clothing feature map of the previous scale to obtain a clothing fusion feature map, and the clothing fusion feature map is transformed according to the cross optical flow field and the cross attention map corresponding to the previous scale to obtain a clothing conversion feature map, and the cross optical flow field and the cross attention map corresponding to the current scale are obtained according to the clothing conversion feature map; the previous scale is smaller than the current scale.

3. The method according to claim 2, characterized in that The method further comprises: When the self-optical flow field and the self-attention map corresponding to the current scale are obtained, if the current scale is not the minimum scale, the self-optical flow field and the self-attention map corresponding to the current scale are obtained according to the human feature map of the current frame of the current scale.

4. The method according to claim 2, characterized in that: The method further comprises: When the cross optical flow field and the cross attention map corresponding to the current scale are obtained, if the current scale is the minimum scale, the cross optical flow field and the cross attention map corresponding to the current scale are obtained according to the clothing feature map of the current scale.

5. The method according to claim 2, characterized in that: Using the target optical flow field and the target attention map as guidance information, the current frame human body image and clothing image are processed to obtain a preliminary fitting result of the current frame human body image, including: Shallowly encode the human body image and clothing image of the current frame respectively to obtain a shallowly encoded human body feature map and a shallowly encoded clothing feature map of the current frame; Perform deformation attention distortion according to the target self-optical flow field, the target self-attention map and the shallowly encoded human feature map of the current frame to obtain a target human clothing feature map of the current frame; Performing deformation attention distortion according to the target cross optical flow field, the target cross attention map and the shallowly encoded clothing feature map to obtain a target clothing feature map; Merge the target human clothing feature map and the target clothing feature map of the current frame to obtain a merged feature map of the current frame; The merged feature map of the current frame is shallowly decoded to obtain a preliminary fitting result of the human body image of the current frame.

6. The method according to any one of claims 1 to 5, characterized in that: According to the similarity between the key feature map of the current frame and the key feature map of the historical frame, the weighted sum of the feature map of the historical frame value is performed to obtain the optimized fitting result of the human body image of the current frame, including: According to the relative size of the similarity between the current frame key feature graph and each historical frame key feature graph, the relative size of the weight assigned to each historical frame value feature graph is determined to obtain the weight of each historical frame value feature graph; the greater the similarity, the greater the weight; According to the weights of the feature maps of each historical frame value, the feature maps of each historical frame value are weighted and summed to obtain the optimized fitting result of the current frame human body image.

7. A device for virtually trying on clothing, characterized in that: The device comprises: The human body image acquisition module is used to obtain multiple frames of human body images in the video; An initial fitting processing module is used to obtain a preliminary fitting result for each frame of the human body image based on the clothing image and each frame of the human body image; An optimized fitting processing module is used to optimize the preliminary fitting result of each frame of the human body image to obtain an optimized fitting result of each frame of the human body image; Wherein, when obtaining the preliminary fitting result of the current frame human body picture, the initial fitting processing module is further used to: obtain the current frame human body feature map and clothing feature map of multiple scales based on the current frame human body picture and clothing picture, optimize the optical flow field and the attention map according to the current frame human body feature map and clothing feature map from small scale to large scale, obtain the target optical flow field and the target attention map, use the target optical flow field and the target attention map as guidance information, process the current frame human body picture and the clothing picture, and obtain the preliminary fitting result of the current frame human body picture; Among them, when optimizing the preliminary try-on results of the current frame human body picture, the optimized try-on processing module is also used to: obtain the preliminary try-on results of the current frame human body picture, the preliminary try-on results of the historical frame human body picture and the optimized try-on results of the historical frame human body picture; obtain the current frame key feature map according to the preliminary try-on results of the current frame human body picture; obtain the historical frame key feature map according to the preliminary try-on results of the historical frame human body picture; obtain the historical frame value feature map according to the optimized try-on results of the historical frame human body picture; according to the similarity between the current frame key feature map and the historical frame key feature map, perform weighted summation on the historical frame value feature map to obtain the optimized try-on result of the current frame human body picture.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images