Video data processing method and device, equipment and storage medium

Abstract

The invention provides a video data processing method and device, equipment and a storage medium, which can be applied to various fields such as electronic shopping and video reloading. The method comprises the following steps: acquiring an image comprising preset clothes and original video data comprising a video object, wherein the original video data comprises N image frames; performing feature extraction on preset clothes in the image to obtain first feature information of the preset clothes; respectively masking the areas to be reloaded in the N image frames to obtain N clothing mask graphs; fusing the N costume mask patterns with one of the N input noise patterns to obtain N fused images; first feature information of preset clothes is used as a control condition of a noise prediction model, noise in the N fused images is predicted through the noise prediction model, N noise prediction values are obtained, de-noising processing is performed on the N input noise maps based on the N noise prediction values, N de-noised images are obtained, updated video data are generated, and the updated video data are sent to the user. And the video reloading effect is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video data processing method, apparatus, device, and storage medium. Background Technology

[0002] With the development of e-shopping and other fields, virtual try-on technology has emerged. The main goal of virtual try-on is to generate a video image of an object wearing preset clothing, ensuring that the fine details of the clothing are preserved and that it blends seamlessly with the surrounding environment.

[0003] Current virtual try-on technology typically combines images of preset clothing with images of the subject to generate an image of the subject wearing the preset clothing. However, current virtual try-on technology only generates images of the changed clothes, and the actual change effect does not meet the expected requirements. Summary of the Invention

[0004] This application provides a video data processing method, apparatus, device, and storage medium that can realize video dress-up, thereby improving the effect of virtual dress-up.

[0005] In a first aspect, this application provides a video data processing method, including:

[0006] Acquire images including preset clothing, and raw video data including video objects, wherein the raw video data includes N image frames;

[0007] Feature extraction is performed on the preset clothing in the image including the preset clothing to obtain the first feature information of the preset clothing;

[0008] The regions to be dressed up in the N image frames are masked separately to obtain N clothing mask images, wherein the regions to be dressed up are at least a portion of the clothing regions of the video object in the N image frames;

[0009] The N clothing mask images are fused with one of the N input noise images to obtain N fused images;

[0010] Using the first feature information of the preset clothing as the noise prediction model denoising control condition, the noise in the N fused images is predicted by the noise prediction model to obtain N noise prediction values;

[0011] The N noise prediction values ​​are used to denoise the N input noise maps to obtain N denoised images, which are then used to generate updated video data containing video objects.

[0012] Secondly, this application provides a model training method, including:

[0013] Obtain training samples, which include training images and training video data. The training images are images including preset clothing, and the video objects in the training video data are wearing the preset clothing. The training video data includes N image frames, where N is a positive integer.

[0014] Feature extraction is performed on the preset clothing in the training images to obtain the first feature information of the preset clothing;

[0015] Mask the areas to be changed into clothing in the N image frames respectively to obtain N clothing mask images, wherein the areas to be changed into clothing are at least a portion of the clothing areas of the video object in the N image frames;

[0016] The N clothing mask images are fused with one of the N input noise images to obtain N fused images; the first feature information of the preset clothing is used as the control condition of the noise prediction model, and the noise in the N fused images is predicted by the noise prediction model to obtain N noise prediction values.

[0017] Based on the N preset noise values ​​and the noise values ​​corresponding to the N input noise maps, the loss of the noise prediction model is determined, and the noise prediction model is trained based on the loss to obtain the trained noise prediction model.

[0018] Thirdly, this application provides a video data processing apparatus, comprising:

[0019] The acquisition unit is used to acquire images including preset clothing and raw video data including video objects, wherein the raw video data includes N image frames, where N is a positive integer;

[0020] An extraction unit is used to extract features from the image including the preset clothing to obtain the first feature information of the preset clothing.

[0021] A masking unit is used to mask the areas to be dressed up in the N image frames respectively to obtain N clothing mask images, wherein the areas to be dressed up are at least a portion of the clothing areas of the video object in the N image frames;

[0022] The fusion unit is used to fuse the N clothing mask images with one of the N input noise images to obtain N fused image images;

[0023] The prediction unit is used to use the first feature information of the preset clothing as the control condition of the noise prediction model, and to predict the noise in the N fused images through the noise prediction model to obtain N noise prediction values.

[0024] The denoising unit is used to denoise the N input noise maps based on the N noise prediction values ​​to obtain N denoised images, which are used to generate updated video data containing video objects.

[0025] In some embodiments, the prediction unit is specifically used to convert the N fused images from pixel space to latent vector space to obtain feature information of the N fused images; determine the input information of the noise prediction model based on the feature information of the N fused images; and use the first feature information of the preset clothing as the control condition of the noise prediction model to process the input information through the noise prediction model to obtain the N noise prediction values.

[0026] In some embodiments, the prediction unit is specifically configured to perform keypoint detection on each of the N image frames to obtain keypoint pose information of the video object in each image frame; determine skeleton feature information of the video object in each image frame based on the keypoint pose information of the video object in each image frame; and determine the input information based on the skeleton feature information of the video object in each image frame and the feature information of each fused image.

[0027] In some embodiments, the prediction unit is specifically configured to, for each image frame, add the skeleton feature information of the video object in the image frame and the feature information of the fused image corresponding to the image frame to obtain the model input information of the image frame; and determine the model input information of the N image frames as the input information.

[0028] In some embodiments, the noise prediction model includes M first network modules, where M is a positive integer greater than 1. The prediction unit is specifically configured to, for the i-th first network module among the M first network modules, obtain the (i-1)th feature information corresponding to the N image frames output by the (i-1)th first network module, where i is a positive integer from 1 to M. If i equals 1, then the (i-1)th feature information corresponding to the N image frames is the input information. Using the first feature information of the preset clothing as a control condition, the i-th first network module processes the (i-1)th feature information corresponding to the N image frames to obtain the i-th feature information corresponding to the N image frames output by the i-th first network module. This process is repeated to obtain the M-th feature information corresponding to the N image frames output by the M-th first network module, and based on the M-th feature information corresponding to the N image frames, obtain the N noise prediction values.

[0029] In some embodiments, the first feature information of the preset clothing is extracted by a clothing feature extraction model, which includes M second network modules. The M second network modules are connected one-to-one with the M first network modules. The extraction unit is specifically used to extract the detailed features and spatial features of the preset clothing in the image including the preset clothing through the M second network modules to obtain the first feature information of the preset clothing corresponding to each of the M second network modules. The prediction unit is specifically used to obtain the i-th first feature information of the preset clothing corresponding to the i-th second network module. Using the i-th first feature information of the preset clothing as a denoising control condition, the i-th first network module processes the (i-1)-th feature information corresponding to the N image frames to obtain the i-th feature information corresponding to the N image frames.

[0030] In some embodiments, the i-th first network module includes a first attention layer, the i-th second network module includes a second attention layer, and the i-th first feature information of the preset clothing is the input information of the second attention layer included in the i-th second network module; the prediction unit is specifically used to obtain the input information of the first attention layer based on the (i-1)-th feature information corresponding to the N image frames; fuse the input information of the first attention layer and the input information of the second attention layer to obtain fused input information; perform intra-frame attention processing on the fused input information corresponding to each image frame in the fused input information through the first attention layer to obtain the first attention processing result corresponding to the N image frames output by the i-th first network module; and determine the i-th feature information corresponding to the N image frames based on the first attention processing result corresponding to the N image frames.

[0031] In some embodiments, the prediction unit is specifically used to perform inter-frame three-dimensional convolution processing on the (i-1)th feature information corresponding to the N image frames to obtain the convolution result; and to determine the input information of the first attention layer based on the convolution result.

[0032] In some embodiments, the prediction unit is specifically configured to fuse the key in the input information of the first attention layer with the key in the input information of the second attention layer to obtain a new key, and to fuse the value in the input information of the first attention layer with the value in the input information of the second attention layer to obtain a new value; and to determine the new key, the query of the first attention layer and the new value as the fused input information.

[0033] In some embodiments, the prediction unit is specifically used to perform inter-frame cross-attention processing on the first attention processing results corresponding to the N image frames to obtain the second attention processing results corresponding to the N image frames; and based on the first attention processing results and the second attention processing results corresponding to the N image frames, to obtain the i-th feature information corresponding to the N image frames.

[0034] In some embodiments, the prediction unit is further configured to extract second feature information of the preset clothing from the image including the preset clothing; and to perform denoising processing on the N fused images through the noise prediction model using the first feature information and the second feature information of the preset clothing as denoising control conditions to obtain the N noise prediction values.

[0035] In some embodiments, the prediction unit is further configured to extract global features of the preset clothing in the image including the preset clothing to obtain a second feature information of the preset clothing; and use the first feature information and the second feature information of the preset clothing as control conditions for the noise prediction model to predict noise in the N fused images through the noise prediction model to obtain the N noise prediction values.

[0036] Fourthly, this application provides a model training apparatus, comprising:

[0037] An acquisition unit is used to acquire training samples, the training samples including training images and training video data, the training images being images including preset clothing, and the video objects in the training video data wearing the preset clothing;

[0038] An extraction unit is used to extract features from a preset garment in the training image to obtain the first feature information of the preset garment.

[0039] The masking unit is used to mask the clothing-changing area in the N image frames included in the training video data to obtain N clothing mask images, where N is a positive integer;

[0040] A fusion unit is used to fuse the N clothing mask images with one of the N input noise images to obtain N fused images;

[0041] The prediction unit is used to use the first feature information of the preset clothing as the control condition of the noise prediction model, and to predict the noise in the N fused images through the noise prediction model to obtain N noise prediction values.

[0042] The training unit is used to determine the loss of the noise prediction model based on the N preset noise values ​​and the noise values ​​corresponding to the N input noise maps, and to train the noise prediction model based on the loss to obtain the trained noise prediction model.

[0043] In some embodiments, the prediction unit is specifically used to convert the N fused images from pixel space to latent vector space to obtain feature information of the N fused images; based on the feature information of the N fused images, determine the input information of the noise prediction model; and use the first feature information of the preset clothing as the control condition of the noise prediction model to process the input information through the noise prediction model to obtain the N noise prediction values.

[0044] In some embodiments, the prediction unit is specifically used to perform keypoint detection on each of the N image frames to obtain keypoint pose information of the video object in each image frame; to encode the keypoint pose information of the video object in each image frame using a pose coding model to obtain skeleton feature information of the video object in each image frame; and to determine the input information based on the skeleton feature information of the video object in each image frame and the feature information of each fused image. The training unit is specifically used to synchronously train the noise prediction model and the pose coding model based on the loss to obtain the trained noise prediction model and the trained pose coding model.

[0045] In some embodiments, the extraction unit is specifically used to extract features from preset clothing in the training image using a clothing feature extraction model to obtain first feature information of the preset clothing; the training unit is specifically used to simultaneously train the noise prediction model and the clothing feature extraction model based on the loss to obtain the trained noise prediction model and the trained clothing feature extraction model.

[0046] In some embodiments, the extraction unit is specifically used to extract features from preset clothing in the training image using a clothing feature extraction model to obtain first feature information of the preset clothing; the training unit is specifically used to simultaneously train the noise prediction model and the clothing feature extraction model based on the loss to obtain the trained noise prediction model and the trained clothing feature extraction model.

[0047] Fifthly, this application provides an electronic device including a processor and a memory. The memory is used to store a computer program, and the processor is used to invoke and run the computer program stored in the memory to perform the methods described in the first or second aspect above.

[0048] In a sixth aspect, a chip is provided for implementing the methods of various implementations of the first aspect described above. Specifically, the chip includes a processor for retrieving and running a computer program from a memory, causing a device equipped with the chip to perform the methods of the first or second aspect described above.

[0049] In a seventh aspect, a computer-readable storage medium is provided for storing a computer program that causes a computer to perform the methods described in the first or second aspect.

[0050] Eighthly, a computer program product is provided, including computer program instructions that cause a computer to perform the methods described in the first or second aspect.

[0051] Ninthly, a computer program is provided that, when run on a computer, causes the computer to perform the methods of the first or second aspect described above.

[0052] In summary, this application acquires images including preset clothing and raw video data including video objects, the raw video data comprising N image frames. Next, features are extracted from the preset clothing in the images including the preset clothing to obtain first feature information of the preset clothing. The area to be dressed in each of the N image frames is masked to obtain N clothing mask images, and each of the N clothing mask images is fused with one of the N input noise images to obtain N fused images. Then, using the first feature information of the preset clothing as the control condition of the noise prediction model, the noise in the N fused images is predicted by the noise prediction model to obtain N noise prediction values. These N noise prediction values ​​are then used to denoise the N input noise images to obtain N denoised images, which are used to generate updated video data containing video objects. Therefore, this application embodiment treats the N fused images corresponding to the raw video data as a time series and inputs them into the noise prediction model. This allows the noise prediction model to focus not only on the internal information of the video frames but also on the relevant information between the video frame sequences during the noise prediction process, resulting in more continuous final video data after the clothing change. In addition, in the process of using the noise prediction model to predict noise in N fused images in this embodiment, not only the first feature information of the preset clothing is considered, but also the relevant information of the clothing mask image. The clothing mask image includes other information in the image frame except for the area to be changed, so that the final generated denoised image not only includes the preset clothing, but also retains other original information in the image frame, thereby improving the effect of video changing clothes. Attached Figure Description

[0053] Figure 1 A schematic diagram of an implementation environment provided for an embodiment of this application;

[0054] Figure 2 A schematic flowchart illustrating a model training method provided in an embodiment of this application;

[0055] Figure 3 This is a schematic diagram illustrating one application scenario of this application;

[0056] Figure 4 This is a schematic diagram of a training sample;

[0057] Figure 5 A schematic diagram for extracting the first feature information of a preset garment;

[0058] Figure 6 A schematic diagram of a clothing feature extraction model;

[0059] Figure 7 This is a schematic diagram of the raw video data;

[0060] Figure 8 A schematic diagram illustrating the pose information of key points in a video object;

[0061] Figure 9 This is an example of a clothing mask image;

[0062] Figure 10 This is an example diagram of noise reduction for a noise prediction model;

[0063] Figure 11 A schematic diagram for extracting feature information from a fused image;

[0064] Figure 12 A schematic diagram for determining input information;

[0065] Figure 13 A schematic diagram of a noise prediction model;

[0066] Figure 14 This is a schematic diagram of a first network model.

[0067] Figure 15 A schematic diagram illustrating the data interaction between a clothing feature extraction model and a noise prediction model;

[0068] Figure 16 This is another schematic diagram of the first network module;

[0069] Figure 17 This is a schematic diagram of data interaction between the first network module and the second network module.

[0070] Figure 18 This is another structural diagram of the first network model;

[0071] Figure 19This is a schematic diagram that uses the second feature information of a pre-set garment as a control condition.

[0072] Figure 20 A schematic flowchart illustrating a video data processing method provided in an embodiment of this application;

[0073] Figure 21 This is a diagram illustrating a video dress-up process.

[0074] Figure 22 This is a schematic diagram of iterative noise reduction during a video costume change process;

[0075] Figure 23 This is a schematic block diagram of a video data processing apparatus provided in an embodiment of this application;

[0076] Figure 24 This is a schematic block diagram of a model training apparatus provided in an embodiment of this application;

[0077] Figure 25 This is a schematic block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0078] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0079] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. In embodiments of the invention, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0080] The image processing method provided in this application can be applied to various fields such as e-shopping and online games, and can realize virtual video dress-up, thereby improving the effect of virtual dress-up.

[0081] To facilitate understanding of the embodiments of this application, the relevant concepts involved in the embodiments of this application will first be introduced:

[0082] Current virtual clothing-changing technologies are all image-based. For example, they acquire an image of clothing and an image of an object, ultimately generating an image of an object wearing the clothing. Another approach is to acquire a video sequence, break it down into a series of consecutive image frames, apply an image-based clothing-changing algorithm to each frame, and finally reassemble the replaced image frames into a video. However, current image-based virtual clothing-changing methods suffer from poor clothing-changing effects.

[0083] To address the aforementioned technical problems, this application proposes a video data processing method. The method acquires an image including a preset garment and raw video data including a video object, the raw video data comprising N image frames. Next, features are extracted from the preset garment in the image including the preset garment to obtain first feature information of the preset garment. The area to be dressed in each of the N image frames is masked to obtain N garment mask images. These N garment mask images are then fused with one of N input noise images to obtain N fused images. Furthermore, using the first feature information of the preset garment as the control condition for a noise prediction model, the noise in the N fused images is predicted by the noise prediction model to obtain N noise prediction values. These N noise prediction values ​​are then used to denoise the N input noise images to obtain N denoised images. These denoised images are used to generate updated video data containing the video object. Therefore, it can be seen that when the N fused images corresponding to the original video data are input into the noise prediction model as a time series in this embodiment, the noise prediction model not only focuses on the internal information of the video frames but also on the relevant information between the video frame sequences during the noise prediction process, making the final generated video data after the costume change more continuous. In addition, in the process of using the noise prediction model to perform noise prediction on the N fused images, this embodiment considers not only the first feature information of the preset clothing but also the relevant information of the clothing mask image. The clothing mask image includes other information in the image frame besides the area to be changed, such as the pose of the video object and the environment in which the video object is located. This ensures that the final generated denoised image not only includes the preset clothing but also retains other original information in the image frame, thereby improving the effect of the video costume change.

[0084] The implementation environment of the image processing method provided in the embodiments of this application is described below.

[0085] Figure 1 This is a schematic diagram of an implementation environment provided for an embodiment of this application, such as... Figure 1 As shown, the implementation environment includes: terminal device 101 and server 102.

[0086] The terminal device 101 is connected to the server 102 via wired or wireless means.

[0087] In some embodiments, the terminal device 101 described above is equipped with a client for the video dress-up platform, and the server 102 can be understood as the server-side or backend of the video dress-up platform. An object (e.g., a user) can interact with the client through the terminal device 101, for example, by inputting an image including preset clothing on the client, and / or inputting raw video data including video objects on the client. The server 102 stores a noise prediction model used to generate the video data after the dress-up.

[0088] In some embodiments, before using the noise prediction model for video dress-up, the noise prediction model is first trained. Specifically, server 102 acquires a training dataset, which includes multiple training samples. Each training sample includes a training image and training video data. The training image is an image of a certain garment, such as an image including a preset garment, and the training video data is video data of a video object wearing the garment. For each training sample in the training dataset, server 102 extracts features from the preset garment in the training image included in the training sample to obtain the first feature information of the preset garment. Simultaneously, the dressing-up area of ​​each of the N image frames in the training video data included in the training sample is masked to obtain N garment mask images. The N garment mask images are then fused with one of the N input noise images to obtain N fused images. Furthermore, using the first feature information of the preset clothing as the control condition of the noise prediction model, the noise in N fused images is predicted by the noise prediction model to obtain N noise prediction values. Then, based on the N noise prediction values ​​and the noise values ​​corresponding to the N input noise images, the loss of the noise prediction model is determined. Based on this loss, the noise prediction model is trained to obtain the trained noise prediction model.

[0089] In this embodiment, after obtaining the trained noise prediction model based on the above steps, the server 102 can save the trained noise prediction model locally on the server. Optionally, the server 102 can also send the trained noise prediction model to the terminal device 101.

[0090] In this embodiment of the application, during video costume change, the trained noise prediction model described above can be used to execute the video data processing method of this embodiment of the application.

[0091] In some embodiments, the video data processing method provided in this application is executed by server 102. For example, an object starts a client of a video dress-up platform on a terminal device 101. The client displays an input box containing an image of a preset outfit and an input box containing raw video data. The object inputs the image of the preset outfit and the raw video data containing the video object into the corresponding input boxes. The terminal device 101 sends the image of the preset outfit and the raw video data to server 102. Server 102 extracts features from the preset outfit in the image containing the preset outfit to obtain first feature information of the preset outfit; simultaneously, server 102 masks the dress-up area of ​​each of the N image frames included in the raw video data to obtain N outfit mask images, and fuses the N outfit mask images with one of the N input noise images to obtain N fused images. Next, server 102 uses the first feature information of the preset clothing as the control condition for the noise prediction model. The noise prediction model predicts noise in N fused images, obtaining N noise prediction values. Then, based on these N noise prediction values, it performs denoising processing on each of the N input noise images to obtain N denoised images. These denoised images are used to generate updated video data containing the video object. Finally, server 102 sends the changed video data to terminal device 101, which displays the changed video data.

[0092] In some embodiments, the video data processing method provided in this application is executed by a terminal device 101. In this case, the terminal device 101 can be any terminal device capable of providing video dress-up services. The server 102 can be any server capable of providing data download services. In this embodiment, the server 102 stores a trained noise prediction model. A client of the video dress-up platform is installed on the terminal device 101. This client displays an input box containing an image of a preset outfit and an input box containing raw video data. The user inputs an image of the preset outfit and raw video data containing the video object into the corresponding input box. In response to the user's input of the image of the preset outfit and raw video data, the terminal device 101 sends a model download request to the server 102. Based on the model download request from the terminal device 101, the server 102 sends the trained noise prediction model to the terminal device 101. The terminal device 101 extracts features from the preset outfit in the image containing the preset outfit to obtain first feature information of the preset outfit; simultaneously, the terminal device 101 masks the area to be dressed up in each of the N image frames included in the raw video data to obtain N outfit mask images. Next, terminal device 101 fuses each of the N clothing mask images with one of the N input noise images to obtain N fused images. Using the preset first feature information of the clothing as the control condition for the noise prediction model, the noise prediction model predicts the noise in the N fused images to obtain N noise prediction values. Then, based on the N noise prediction values, the N input noise images are denoised to obtain N denoised images. These N denoised images are used to generate updated video data containing the video object. Finally, terminal device 101 displays the video data after the clothing change.

[0093] In some embodiments, the terminal device 101 includes, but is not limited to, desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices may include smartwatches, smart bracelets, and head-mounted devices. Terminal devices are often equipped with a display device, which may be a monitor, display screen, touchscreen, etc., and the touchscreen may be a touchscreen, touch panel, etc.

[0094] In some embodiments, the server 102 described above can be one or more servers. When there are multiple servers, at least two servers are used to provide different services, and / or at least two servers are used to provide the same service, such as providing the same service in a load-balanced manner. This application embodiment does not limit this. The server described above can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server can also be a node in a blockchain.

[0095] It should be noted that the implementation environment of this application embodiment includes, but is not limited to, Figure 1 As shown.

[0096] The technical solutions of the embodiments of this application will be described in detail below through some examples. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0097] This application's embodiments mainly relate to video clothing change, that is, replacing the clothing of a video object in the original video data with preset clothing, to obtain video data of the video object wearing the preset clothing. For example... Figure 2 As shown, the video object in the original video data is dressed in the preset clothing shown in Figure 1, resulting in video data of the video object wearing the preset clothing.

[0098] This application uses a noise prediction model for video costume change. Before using the noise prediction model for video costume change, the noise prediction model needs to be trained first.

[0099] The following is combined Figure 3 The training process of the noise prediction model in the embodiments of this application will be introduced.

[0100] Figure 3 This is a schematic flowchart illustrating a model training method provided in an embodiment of this application. The execution entity of this embodiment can be a device with model training functionality, such as a model training apparatus. This model training apparatus can be the one described above. Figure 1 The server 102 or terminal device 101 shown can also be Figure 1 The system consists of server 102 or terminal device 101 as shown. For ease of description, the following embodiments use electronic devices as the execution subject to illustrate the model training method of this application.

[0101] like Figure 2 As shown, the model training method in this application embodiment includes:

[0102] S101. Obtain training samples.

[0103] The training samples consist of training images and training video data. The training images include images of preset clothing, and the video data contains video objects wearing the preset clothing.

[0104] It should be noted that the data used in the embodiments of this application and the process of obtaining this data comply with the relevant laws and regulations.

[0105] Before training a noise prediction model, an electronic device first acquires a training dataset. This dataset includes multiple training samples, each consisting of a training image and a training video. The training image is an image featuring a specific garment, such as a preset garment. The training video is video data of an object wearing that garment (e.g., the preset garment). The training image is typically a tiled image of the garment or the image containing the most garment features.

[0106] For example, the training dataset of this application embodiment is shown in Table 1:

[0107] Table 1

[0108]

[0109] It should be noted that the objects in different training video data can be the same object or different objects, and this application embodiment does not impose any restrictions on this.

[0110] The objects in this application's embodiments can be real biological entities, such as humans or animals. Optionally, the objects in this application's embodiments can also be virtual objects, such as virtual humans or virtual animals. This application's embodiments do not limit the specific type of video object.

[0111] In this embodiment, the noise prediction model is trained multiple times using a training dataset. In each round of training, a batch of training samples from the training dataset is selected for training. The process of training the noise prediction model using each training sample from the corresponding batch is essentially the same in each round. For ease of description, a single training sample is used as an example. For instance, the model training process of this embodiment is described using training images including preset clothing and training video data showing a video object wearing the preset clothing.

[0112] S102. Extract features from the preset clothing in the training image to obtain the first feature information of the preset clothing.

[0113] In the embodiments of this application, such as Figure 4 As shown, when an electronic device trains a noise prediction model using training images of preset clothing and training video data of people wearing the preset clothing, it first extracts the first feature information of the preset clothing from the training images.

[0114] This application does not limit the specific method by which the electronic device extracts the first feature information of a preset garment in the training image.

[0115] In some embodiments, the electronic device uses a clothing feature extraction model to extract first feature information of a preset clothing in the training image.

[0116] This application does not limit the specific network structure of the clothing feature extraction model in its embodiments. In one example, the clothing feature extraction model is ReferenceNet.

[0117] In some examples, the above-mentioned clothing feature extraction model is pre-trained. In the model training process of this application embodiment, the clothing feature extraction model is not trained.

[0118] In some examples, the above-mentioned clothing feature extraction model was not pre-trained. In the model training process of this application embodiment, the clothing feature extraction model is trained.

[0119] In one example, the clothing feature extraction model in this application embodiment is a pre-trained diffusion model. This pre-trained diffusion model has been trained on a large number of images and has strong image processing capabilities.

[0120] In some embodiments, the clothing feature extraction model of this application is as follows: Figure 5 As shown, this clothing feature extraction model is used to extract detailed features and spatial details of a preset clothing in a training image to obtain local feature information of the preset clothing. This local feature information is recorded as the first feature information of the target image. The electronic device inputs the training image into this clothing feature extraction model, which is a diffusion model. The main component is a U-shaped network, including M second network modules (…). Figure 5 The image shows six second network modules. Thus, the electronic device uses these M second network modules to extract the detailed and spatial features of the preset clothing in the training image, obtaining the first feature information of the preset clothing corresponding to each of the M second network modules.

[0121] This application does not limit the specific connection method of the M second network modules. In one example, the M second network modules are connected in series. In another example, the M second network modules are connected in parallel. In yet another example, some of the M second network modules are connected in series, and some are connected in parallel.

[0122] This application does not limit the specific network structure of the second network module in its embodiments. In one example, the second network module includes a convolutional layer and an attention layer, denoted as the second convolutional layer and the second attention layer, respectively. For example,... Figure 6 As shown, the electronic device first performs convolution processing on the training image through the second convolutional layer in the second network module 1 to obtain feature information a1. Next, feature information a1 is input into the second attention layer for attention processing to obtain feature information b1. Feature information b1 is then input into the second network module 2, processed through the second convolutional layer and the second attention layer in the second network module 2 to obtain feature information b2. Next, feature information b2 is input into the second network module 3, and this process is repeated until feature information b5 is input into the second network module 6, processed through the second convolutional layer and the second attention layer in the second network module 6 to obtain feature information b6.

[0123] In some embodiments of this application, the feature information output by the last network layer of the above-mentioned clothing feature extraction model is denoted as the first feature information of the preset clothing in the training image. For example, the above-mentioned Figure 5 Feature information b6 in the training image is determined as the first feature information of the preset clothing in the training image. At this time, the first feature information of the preset clothing in the training image includes one feature information.

[0124] In some embodiments of this application, the feature information corresponding to the intermediate layer in the above-mentioned clothing feature extraction model is denoted as a feature information of a preset clothing in the training image, wherein the feature information corresponding to the intermediate layer includes the input information or the output information of the intermediate layer. For example, the above-mentioned Figure 5 The input information of the second attention layer in each second network module is used to determine the first feature information of the preset clothing in the training image. At this time, the first feature information of the preset clothing in the training image includes M feature information.

[0125] In this embodiment of the application, the electronic device extracts the first feature information of the preset clothing included in the training image based on the above steps.

[0126] S103. Mask the areas to be dressed in the N image frames respectively to obtain N clothing mask images.

[0127] The area to be dressed up is at least a portion of the clothing area of ​​the video object in the N image frames. That is, the area to be dressed up can be the entire clothing area of ​​the video object in the image frame, or a portion of the clothing image of the video object in the image frame. For example, if the clothing is a pre-set top, then the area to be dressed up is the top area of ​​the video object in the image frame, where N is a positive integer.

[0128] It should be noted that the execution order of the steps of extracting the first feature information of the preset clothing in the training image and the steps of masking the clothing mask map of each image frame by masking the clothing area to be changed in each image frame are not limited in this embodiment.

[0129] The training video data in this embodiment includes N image frames. This training video data is divided into a series of image frames, denoted as... For example, the training video data includes, Figure 7 The three image frames shown are N equal to 3.

[0130] For each of these N image frames, the electronic device masks the area to be dressed in each image frame, thereby obtaining the clothing mask image for each image frame.

[0131] In the embodiments of this application, the specific method by which the electronic device determines the clothing mask image of each of the N image frames is basically the same. For ease of description, this paper takes one image frame as an example, for example, the j-th image frame in N image frames, where j is a positive integer less than or equal to N.

[0132] This application embodiment does not limit the specific method by which the electronic device masks the clothing mask area of ​​the image frame to obtain the clothing mask image of the image frame.

[0133] In some embodiments, for the j-th image frame out of N image frames, keypoint detection is performed on the j-th image frame to obtain the keypoint pose information of the video object in the j-th image frame. For example, electronic devices can use a human keypoint detection model, such as the OpenPose model, to detect keypoints in the j-th image frame and obtain the keypoint pose information of the video object in that j-th image frame. For example, Figure 8 It shows Figure 7 The three image frames shown correspond to keypoint pose information respectively. The electronic device segments different regions of the video object in the j-th image frame to obtain the region segmentation map of the j-th image frame. For example, the electronic device uses a human segmentation model, such as CIHP_PGN, to segment the j-th image frame. The video object is split into different regions to obtain segmentation maps. Next, the electronic device uses images including preset clothing and segmentation maps of different regions of the video object corresponding to the j-th image frame. and the pose information of key points of video objects in the j-th image frame The required clothing-changing area for the video object in the j-th image frame is determined, i.e., the area to be dressed up. For example, the electronic device may include an image of preset clothing and a segmentation map of different regions of the video object corresponding to the j-th image frame. and the pose information of key points of video objects in the j-th image frame In the input semantic generation module (SGM), the region to be dressed up in the j-th image frame is identified. For example, if the clothing is preset to be a top, the above method can be used to determine the top region of the video object in the j-th image frame as the region to be dressed up. Then, the region to be dressed up in the j-th image frame is masked to obtain the clothing mask image corresponding to the j-th image frame.

[0134] In other words, in this embodiment of the application, the electronic device processes N image frames. Keypoint detection is performed to obtain the pose information of keypoints of the video object in each of N image frames, denoted as . At the same time, the electronic device processes N image frames. The video object is segmented into different regions to obtain segmentation maps of different regions of the video object in each of N image frames, denoted as [missing information]. In this way, the electronic device can perform masking on the area to be dressed up in each image frame based on the key point pose information of the video object in each image frame and the region segmentation map of each image frame. This yields a clothing mask map where the clothing area corresponding to the preset clothing in each image frame is occluded, denoted as [the mask map is missing from the original text]. For example, Figure 9 It shows Figure 7 The clothing mask images corresponding to the three image frames shown.

[0135] S104. Fuse the N clothing mask images with one of the N input noise images to obtain N fused images.

[0136] In this process, a clothing mask image and an input noise image correspond to each other to obtain a fused image.

[0137] In this embodiment, for each of the N clothing mask images, an input noise image is determined. This embodiment does not limit the specific method by which the electronic device determines the input noise image. In one example, the electronic device randomly generates N noise values, denoted as... These N noise values ​​are denoted as N input noise maps. In some examples, the size of these N input noise maps is the same as the size of the N clothing mask maps. For example, for clothing masks... Figure 1 The clothing mask image and the input noise Figure 1 Corresponding. If clothing mask Figure 1 When including m*n pixels, the input noise generated in this embodiment of the application... Figure 1 It also includes m*n noise points, and the noise value of each noise point is... In other words, clothing masking Figure 1 A pixel and input noise Figure 1 One of the noise points corresponds to this.

[0138] Next, the electronic device fuses the N clothing mask images with one of the N input noise images to obtain N fused images.

[0139] In this embodiment, an electronic device fuses N clothing mask images with one of N input noise images to obtain N fused images.

[0140] In one possible implementation, for each of the N clothing mask images, the input noise image corresponding to that clothing mask image is added to that clothing mask image to obtain the fused image corresponding to that clothing mask image.

[0141] In one possible implementation, for each of the N clothing mask images, the corresponding input noise image is concatenated with the clothing mask image along the channel dimension. For example, concatenating each of the N clothing mask images with one of the N input noise images along the channel dimension yields N fused images.

[0142] S105. Using the first feature information of the preset clothing as the control condition of the noise prediction model, the noise in the N fused images is predicted by the noise prediction model to obtain N noise prediction values.

[0143] The embodiments of this application do not impose restrictions on the initial model of the noise prediction model.

[0144] In some embodiments, the initial model for the noise prediction model is a model that has not been trained on video data.

[0145] In some embodiments, the initial model of the video dress-up module is a pre-trained video processing model, which is trained on a large amount of video data and possesses powerful video data processing capabilities. In this embodiment, the pre-trained video processing module is fine-tuned using the training samples to obtain a noise prediction model. This ensures that the trained noise prediction model retains its video processing capabilities while maintaining the dress-up function, thereby improving the effect of the video dress-up.

[0146] In this embodiment, the electronic device extracts the first feature information of preset clothing from the training image, and performs masking processing on the clothing-to-be-changed area of ​​each of the N image frames included in the training video data to obtain N clothing mask images. Then, the N clothing mask images and N input noise images are fused to obtain N fused images. Next, as... Figure 10 As shown, the first feature information of the preset clothing and N fused images are input into the noise prediction model, so that the noise prediction model uses the first feature information of the preset clothing as the control condition to predict the noise in the N fused images and obtain N noise prediction values.

[0147] In this application embodiment, the electronic device uses the first feature information of the preset clothing as the control condition of the noise prediction model, and the noise prediction model is used to predict the noise in N fused images to obtain N noise prediction values. The specific method is not limited.

[0148] In some embodiments, such as Figure 10 As shown, the electronic device directly inputs N fused images as a time series into the noise prediction model, so that the noise prediction model uses the first feature information of the preset clothing as the control condition to predict the noise in the N fused images and obtain N noise prediction values.

[0149] In some embodiments, S105 above includes the following steps S105-A to S105-C:

[0150] S105-A: Transform the N fused images from pixel space to latent vector space to obtain the feature information of the N fused images;

[0151] S105-B: Based on the feature information of N fused images, determine the input information of the noise prediction model;

[0152] S105-C: Using the first feature information of the preset clothing as the control condition of the noise prediction model, the noise prediction model processes the input information to obtain N noise prediction values.

[0153] In this implementation, to improve image processing speed and reduce computational workload, the electronic device can transform N fused images from pixel space to latent vector space to obtain feature information of N fused images. The main purpose of this part is to represent the features of the fused images using high-level semantic features, which not only reduces the spatial dimension of the fused images, thereby improving training speed, but also facilitates better interaction with the control conditions of the noise prediction model.

[0154] In one example, an electronic device uses an image feature encoder to transform N fused images from pixel space to latent vector space to obtain feature information of the N fused images.

[0155] The embodiments of this application do not limit the specific network structure of the image feature encoder.

[0156] In some embodiments, the image feature encoder is a variational autoencoder (VAE). For example, such as... Figure 11 As shown, the electronic device uses a variational autoencoder (VAE) to transform N fused images from pixel space to latent space, obtaining feature information of the N fused images. In some embodiments, the VAE is already trained, and is not trained during the model training process of this application embodiment. In some embodiments, the VAE is trained together with the model during the model training process of this application embodiment.

[0157] Next, the electronic device determines the input information for the noise prediction model based on the feature information of the N fused images.

[0158] This application does not limit the specific method by which the sub-device determines the input information of the noise prediction model based on the feature information of N fused images.

[0159] In some embodiments, the electronic device can directly determine the feature information of N fused images as the input information of the noise prediction model.

[0160] In some embodiments, S105-B above includes the following steps S105-B1 to S105-B3:

[0161] S105-B1: Perform keypoint detection on each of the N image frames to obtain the keypoint pose information of the video object in each image frame;

[0162] S105-B2: Based on the pose information of the key points of the video object in each image frame, determine the skeleton feature information of the video object in each image frame;

[0163] S105-B3: Determine the input information based on the skeleton feature information of the video object in each image frame and the feature information of each fused image.

[0164] In this implementation, to further enhance the video dress-up effect, the pose information of the video object is also considered during the dress-up process. Specifically, the electronic device performs keypoint detection on each of the N image frames to obtain the keypoint pose information of the video object in each image frame. Based on the keypoint pose information of the video object in each image frame, the skeleton feature information of the video object in each image frame is determined. For example, an electronic device can collect the keypoint pose information of the video object from N image frames. The input is processed by a pose encoder to obtain the skeleton feature information of the video object in each of the N image frames. In one example, the electronic device can input the skeleton feature information of the video object in each of the N image frames one by one into the pose encoder, so that the pose encoder outputs the skeleton feature information of the video object in one image frame at a time. In another example, the electronic device can simultaneously input the skeleton feature information of the video object in the N image frames as a sequence into the pose encoder, so that the pose encoder outputs the skeleton feature information of the video object in all N image frames simultaneously.

[0165] In this embodiment of the application, after the electronic device determines the skeleton feature information of the video object in each of the N image frames, such as... Figure 12 As shown, the input information is determined based on the skeleton feature information of the video object in each of the N image frames and the feature information of each fused image.

[0166] In one possible implementation, for each image frame, such as image frame i, the electronic device concatenates the skeleton feature information of the video object in image frame i with the feature information of the fused image i corresponding to image frame i to obtain the model input information of image frame i. Then, the model input information of N image frames is determined as the input information.

[0167] In one possible implementation, for each image frame, such as image frame i, the electronic device adds the skeleton feature information of the video object in image frame i to the feature information of the fused image corresponding to image frame i to obtain the model input information of image frame i. Then, the model input information of N image frames is determined as the input information.

[0168] In some embodiments, in addition to the skeleton feature information of the video object in the image frame and the feature information of the fused image, the above input information may also include other information, such as ambient light and other information that can improve the video dress-up effect. This application embodiment does not limit this.

[0169] In this embodiment, after determining the input information of the noise prediction model based on the above steps, the electronic device inputs the input information and the first feature information of the preset clothing into the noise prediction model, so that the noise prediction model processes the input information with the first feature information of the preset clothing as the control condition to obtain N noise prediction values.

[0170] In some embodiments, the noise prediction model described above is a diffusion model, including forward diffusion and backward diffusion. The forward diffusion process fuses N clothing mask images and N input noise images to obtain N fused images. Backward diffusion estimates the noise and subtracts the estimated noise from the N input noise images to obtain N denoised images. In this embodiment, compared to using pure noise as input information for the noise prediction model, this embodiment uses the N fused images obtained by fusing N clothing mask images and N input noise images as input information for the noise prediction model. This way, during the backward diffusion process, the noise prediction model not only considers the first feature information of the preset clothing to ensure that the area corresponding to the area to be dressed in the image frame accurately represents the preset clothing in the generated denoised image, but also considers the relevant information of the clothing mask images. This allows the generated denoised image to retain the specific details of the original image frame, thereby improving the effect of video dress-up.

[0171] The following describes the specific process in S105-C above, in which the first feature information of the preset clothing is used as the control condition of the noise prediction model, and the noise prediction model processes the input information to obtain N noise prediction values.

[0172] In some embodiments, the electronic device uses the first feature information of the preset clothing as the control condition, processes the input information through a noise prediction model, and obtains N noise prediction values.

[0173] In some embodiments, the noise prediction model described above includes M first network modules, and S105-C includes the following steps S105-C1 to S105-C3:

[0174] S105-C1. For the i-th first network module among the M first network modules, obtain the i-1-th feature information corresponding to the N image frames output by the (i-1)-th first network module, where i is a positive integer from 1 to M. If i equals 1, then the (i-1)-th feature information is the above input information.

[0175] S105-C2: Using the first feature information of the preset clothing as the control condition, the i-th first network module processes the (i-1)th feature information corresponding to N image frames to obtain the i-th feature information corresponding to N image frames output by the i-th first network module.

[0176] S105-C3: Based on the Mth feature information corresponding to the N image frames output by the Mth first network module, N noise prediction values ​​are obtained.

[0177] In the embodiments of this application, such as Figure 13 As shown, the noise prediction model includes M first network modules ( Figure 13 As shown in the case of M (e.g., 6), for each of the M network modules, such as the i-th first network module, the electronic device first obtains the feature information corresponding to the N image frames output by the previous network module of the i-th first network module, that is, the (i-1)-th network module. For ease of description, the feature information corresponding to the N image frames output by the (i-1)-th network module is denoted as the (i-1)-th feature information corresponding to the N image frames.

[0178] In one example, if i equals 1, that is, the i-th first network module is the first first network module in the noise prediction model, then the (i-1)-th feature information corresponding to the N image frames is the input information determined above.

[0179] In one example, if i is greater than 1, then the (i-1)th feature information corresponding to the above N image frames is the feature information corresponding to the N image frames output by the (i-1)th first network module. For example, when the (i-1)th first network module is the first first network module of the video diffusion module, then the first first network module processes the above input information and outputs the (i-1)th feature information corresponding to the N image frames.

[0180] In this embodiment of the application, after the electronic device obtains the (i-1)th feature information corresponding to N image frames, it inputs the (i-1)th feature information corresponding to the N image frames into the i-th first network module, so that the i-th first network module processes the (i-1)th feature information corresponding to the N image frames with the first feature information of the preset clothing as the control condition, and obtains the i-th feature information corresponding to the N image frames output by the i-th first network module.

[0181] This application embodiment does not limit the specific method by which the electronic device uses the first feature information of the preset clothing as the control condition, and processes the (i-1)th feature information corresponding to N image frames through the i-th first network module to obtain the i-th feature information corresponding to N image frames output by the i-th first network module.

[0182] In some embodiments, the electronic device fuses the (i-1)th feature information corresponding to N image frames with the first feature information of a preset garment, for example, by adding or splicing them. The fused feature information is then input into the i-th first network module for processing, resulting in the i-th feature information corresponding to the N image frames output by the i-th first network module. In this implementation, the first feature information of the preset garment input into each of the M first network modules is consistent.

[0183] In some embodiments, the electronic device extracts first feature information of a preset garment from a training image using a garment feature extraction model. For example... Figure 5 As shown, the clothing feature extraction model includes M second network modules, which are connected one-to-one with the M first network modules in the noise prediction model. In this case, the specific method for extracting features from the preset clothing in the training image in S102 to obtain the first feature information of the preset clothing is as follows: Features are extracted from the preset clothing in the training image through the M second network modules to obtain the first feature information of the preset clothing corresponding to each of the M second network modules. In this case, S105-C2 may include the following steps S105-C21 and S105-C22:

[0184] S105-C21. Obtain the i-th first feature information of the preset clothing corresponding to the i-th second network module in the clothing feature extraction model;

[0185] S105-C22. Using the i-th first feature information of the preset clothing as the control condition, the i-th first network module processes the (i-1)-th feature information corresponding to N image frames to obtain the i-th feature information corresponding to N image frames.

[0186] In this implementation, the clothing feature extraction model includes M second network modules, and the noise prediction model includes M first network modules. These M second network modules are connected one-to-one with the M first network modules. For example, taking the i-th first network module among the M first network modules as an example, this i-th first network module is connected to the i-th second network module. Therefore, the electronic device acquires the first feature information of the preset clothing corresponding to the i-th second network module. For ease of description, the first feature information of the preset clothing corresponding to the i-th second network module is denoted as the i-th first feature information of the preset clothing. Thus, as... Figure 14 As shown, the electronic device uses the i-th first feature information of the preset clothing as the control condition, and processes the (i-1)-th feature information corresponding to N image frames through the i-th first network module to obtain the i-th feature information corresponding to N image frames.

[0187] In this embodiment of the application, the electronic device obtains the i-th first feature information of the preset clothing corresponding to the i-th second network module in at least the following ways:

[0188] Example 1: The feature information output by the last network layer of the i-th second network module is denoted as the i-th first feature information of the preset clothing.

[0189] Example 2: The feature information output by the intermediate layer of the i-th second network module is denoted as the i-th first feature information of the preset clothing.

[0190] Example 3: The input feature information of the intermediate layer of the i-th second network module is denoted as the i-th first feature information of the preset clothing.

[0191] Example 4: The input feature information of the first network layer of the i-th second network module is denoted as the i-th first feature information of the preset clothing.

[0192] In some embodiments, the first network module of this application includes an attention layer, denoted as the first attention layer, and the second network module also includes an attention layer, denoted as the second attention layer. It should be noted that the first and second network modules may also include other network layers, such as convolutional layers, etc., and this application does not impose any limitations on this.

[0193] In one example, if the i-th first feature information of the preset clothing is the input information of the second attention layer in the i-th second network module, then the above S105-C22 includes the following steps S105-C221 to S105-C224:

[0194] S105-C221. Based on the (i-1)th feature information corresponding to N image frames, obtain the input information of the first attention layer included in the i-th first network module;

[0195] S105-C222: Fuse the input information of the first attention layer and the input information of the second attention layer to obtain fused input information;

[0196] S105-C223, Self-attention processing is performed on the fused input information corresponding to each image frame in the fused input information through the first attention layer, and the first attention processing results corresponding to the N image frames output by the i-th first network module are obtained.

[0197] S105-C224. Based on the first attention processing results corresponding to N image frames, determine the i-th feature information corresponding to the N image frames.

[0198] In the embodiments of this application, such as Figure 15As shown, the electronic device uses the input information of the second attention layer of each of the M second network modules included in the clothing feature extraction model as the control condition of the corresponding first network module. For example, taking the i-th first network module as an example, the electronic device fuses the input information of the second attention layer in the i-th second network module as the control condition of the i-th first network module with the (i-1)-th feature information corresponding to N image frames, and uses this as the input information of the first attention layer in the i-th first network module.

[0199] Specifically, for the i-th first network module, the electronic device first obtains the input information of the first attention layer based on the (i-1)-th feature information corresponding to the above N image frames.

[0200] This application does not limit the specific method by which the electronic device obtains the input information of the first attention layer based on the (i-1)th feature information corresponding to N image frames.

[0201] In some embodiments, the electronic device directly uses the (i-1)th feature information corresponding to N image frames as the input information of the first attention layer.

[0202] In some embodiments, the electronic device performs inter-frame 3D convolution processing on the (i-1)th feature information corresponding to N image frames to obtain the convolution result, and then determines the input information of the first attention layer based on the convolution result. For example, as... Figure 16 As shown, the first network module in this embodiment further includes a three-dimensional convolutional layer, which is connected in series with the first attention layer. The electronic device first determines the input information of the three-dimensional convolutional layer based on the (i-1)th feature information corresponding to N image frames. For example, the (i-1)th feature information corresponding to the N image frames is converted into a data format and then used as the input information of the three-dimensional convolutional layer. Alternatively, two-dimensional convolution processing is performed on the (i-1)th feature information corresponding to the N image frames, and the two-dimensional convolution result is converted into a data format and then weighted with the (i-1)th feature information corresponding to the N image frames to obtain the input information of the three-dimensional convolutional layer. Next, the input information of the three-dimensional convolutional layer is subjected to inter-frame three-dimensional convolution processing through the three-dimensional convolutional layer to obtain convolution result 1. Then, based on this convolution result 1, the input information of the first attention layer is determined.

[0203] In one example, the convolution result 1 is compared with the key matrix W. K Query matrix W Q Sum matrix W V Multiply the results to obtain the input information for the first attention layer.

[0204] In one example, convolution result 1 is weighted with the input information of the 3D convolutional layer to obtain convolution result 2. Convolution result 2 is then weighted with the key matrix W. KQuery matrix W Q Sum matrix W V Multiply the results to obtain the input information for the first attention layer.

[0205] After determining the input information of the first attention layer, the electronic device fuses the input information of the first attention layer and the input information of the second attention layer to obtain the fused input information.

[0206] This application embodiment does not limit the specific method by which the electronic device fuses the input information of the first attention layer and the input information of the second attention layer to obtain the fused input information.

[0207] In one possible implementation, such as Figure 17 As shown, the electronic device fuses the key K in the input information of the first attention layer with the key K in the input information of the second attention layer to obtain a new key K, and fuses the value V in the input information of the first attention layer with the value V in the input information of the second attention layer to obtain a new value V; the new key K, the query Q of the first attention layer and the new value V are determined as the fused input information.

[0208] Next, the electronic device inputs the aforementioned fused input information into the first attention layer of the i-th first network module. This first attention layer performs intra-frame attention processing on the fused input information corresponding to each image frame, resulting in the first attention processing results for N image frames. This application embodiment does not limit the specific type of the first attention layer.

[0209] In one example, the first attention layer in this embodiment is a self-attention layer. The electronic device performs self-attention processing on the fused input information corresponding to each image frame in the fused input information through the first attention layer, obtaining the first attention processing result corresponding to each of the N image frames. For example, by substituting the new key K, the query Q of the first attention layer, and the new value V into the attention calculation formula... In this process, the first attention processing result corresponding to the image frame is calculated. The first attention processing results corresponding to the N image frames are then concatenated to obtain the first attention processing result corresponding to the N image frames.

[0210] After determining the first attention processing results corresponding to N image frames based on the above steps, the electronic device executes the above steps S105-C224, that is, based on the first attention processing results corresponding to N image frames, it determines the i-th feature information corresponding to N image frames.

[0211] This application does not limit the specific method by which an electronic device determines the i-th feature information corresponding to N image frames based on the first attention processing results corresponding to N image frames.

[0212] In one possible implementation, the electronic device uses the first attention processing results corresponding to N image frames to determine the i-th feature information corresponding to the N image frames.

[0213] In one possible implementation, the electronic device performs inter-frame cross-attention processing on the first attention processing results corresponding to N image frames to obtain a second attention processing result corresponding to N image frames; based on the first and second attention processing results corresponding to N image frames, the i-th feature information corresponding to N image frames is obtained. The inter-frame cross-attention processing of this application can be understood as inter-frame self-attention processing, that is, processing the first attention feature information corresponding to N image frames in the time dimension to obtain the second attention feature information corresponding to N image frames.

[0214] For example, such as Figure 18 As shown, the first network module in this embodiment further includes a third attention layer, which is a temporal attention layer. This third attention layer is connected to the first attention layer. The third attention layer performs inter-frame cross-attention processing on the information of each image frame in the first attention processing result corresponding to N image frames, completing information interaction between frames. As described above, the first attention layer performs intra-frame self-attention processing on the fused input information corresponding to each image frame, understanding the information of each frame to obtain the first attention processing result corresponding to N image frames. Then, the first attention processing result corresponding to the N image frames is input into the third attention layer. The third attention layer then interactively understands the information at the same position between different frames in the first attention processing result corresponding to the N image frames to obtain the second attention processing result corresponding to the N image frames. For example, if the third attention layer is a temporal attention layer, the electronic device inputs the first attention processing result corresponding to the N image frames into the temporal attention layer, performs self-attention processing on the information corresponding to different image frames at the same position point, and obtains the second attention processing result corresponding to the N image frames.

[0215] Finally, based on the first attention processing results and the second attention processing results corresponding to the N image frames, the i-th feature information corresponding to the N image frames is obtained. For example, the first attention processing results and the second attention processing results are weighted to obtain the i-th feature information corresponding to the N image frames output by the i-th first network module.

[0216] Next, the electronic device inputs the i-th feature information corresponding to the N image frames into the (i+1)-th first network module, repeating the above steps until the M-th first network module outputs the M-th feature information corresponding to the N image frames. Then, based on the M-th feature information corresponding to the N image frames, N noise prediction values ​​are determined. For example, the M-th feature information corresponding to the N image frames is processed to obtain N noise prediction values.

[0217] In some embodiments, to further enhance the effect of video dress-up, the electronic device further extracts global features of preset clothing in the training image to obtain second feature information of the preset clothing. In this embodiment, extracting global features of the preset clothing includes at least features such as color, shape, and texture. This embodiment does not limit the specific method of extracting the second feature information of the preset clothing. In one possible implementation, the electronic device uses an image feature extraction model, such as a Contrastive Language-Image Pre-Training (CLIP) image encoder, to extract global features of the preset clothing in the training image, obtaining a one-dimensional vector feature (e.g., a 1x768 vector), and denotes this one-dimensional vector feature as the second feature information of the preset clothing. Then, as... Figure 19 As shown, the first and second feature information of the preset clothing are used as control conditions for the noise prediction model. The noise prediction model predicts noise in N fused images, resulting in N noise prediction values. Since the second feature information of the preset clothing reflects the global features of the preset clothing, and the first feature information reflects the local detail features of the preset clothing, using the first and second feature information of the preset clothing as control conditions for the noise prediction model can improve the accuracy of noise prediction. Furthermore, when denoising the N input noise images based on the accurately predicted noise values, the denoising effect of the generated N denoised images can be improved, thereby enhancing the dressing-up effect in the video. This application embodiment does not limit the fusion method of the second feature information of the preset clothing with the feature information corresponding to the M first network modules included in the noise prediction model. For example, the second feature information of the preset clothing can be fused with the input information of all or part of the first network modules in the M first network modules, or with the output information of all or part of the first network modules in the M first network modules, or with the intermediate information of all or part of the first network modules in the M first network modules. The specific method can be determined according to actual needs. In one example, the second feature information of the preset clothing is input into the third convolutional layer of the first network module.

[0218] After the electronic device obtains N noise prediction values ​​based on the above steps, it executes the following step S105.

[0219] S106. Based on N noise prediction values ​​and the noise values ​​corresponding to N input noise maps, determine the loss of the noise prediction model, and train the noise prediction model based on the loss to obtain the trained noise prediction model.

[0220] In step S103 above, the N input noise maps in this embodiment correspond to N noise values, denoted as... Next, the electronic device uses the first feature information of the preset clothing as the control condition for the noise prediction model. The noise prediction model then predicts the noise in the N fused images, obtaining N predicted noise values. Based on these N predicted noise values ​​and the N predicted noise values, the electronic device can determine the loss of the noise prediction model. For example, the average difference between the N predicted noise values ​​and the N predicted noise values ​​can be used as the loss of the noise prediction model. Furthermore, based on the model's loss, the parameters in the noise prediction model are adjusted to obtain the trained noise prediction model.

[0221] In some embodiments, if the above-mentioned electronic device processes the key point pose information of the video object in each image frame through the pose coding model to determine the skeleton feature information of the video object in each image frame, and the pose coding model is an untrained model, then in this embodiment of the application, the electronic device can perform end-to-end training on the noise prediction model and the pose coding model based on the loss of the above model to obtain the trained noise prediction model and the trained pose coding model.

[0222] In some embodiments, if the electronic device extracts features from the target image in the training image using the clothing feature extraction model to obtain the first feature information of the preset clothing, and the clothing feature extraction model is not trained, the electronic device can perform end-to-end training on the noise prediction model and the clothing feature extraction model based on the loss of the model to obtain the trained noise prediction model and the trained clothing feature extraction model.

[0223] The above embodiments describe a training process. In this embodiment, for a training sample, noise can be added K times, and after the above steps, denoising is performed K times through the noise prediction model. The model loss corresponding to each denoising is determined, and then the parameters in the noise prediction model are updated K times using the model loss determined in each denoising process. Then, using new training samples, the model is updated multiple times to finally obtain the trained noise prediction model.

[0224] For example, using the training samples above, assume K equals 20. First, the electronic device generates N noise values ​​of 1, denoted as... Based on these N noise values, N input noise values ​​are obtained. Figure 1 This will reduce the number of input noises to N. Figure 1 The image is fused with N clothing mask images to obtain N fused images 1. Using the first feature information of the preset clothing as a control condition, the N fused images 1 are denoised using a noise prediction model to obtain N noise prediction values ​​1. Based on these N noise prediction values ​​1 and the aforementioned N noise values ​​1, the model loss 1 is determined, and the noise prediction model is updated based on the loss 1 to obtain noise prediction model 1. Next, the electronic device generates N noise values ​​2, denoted as... Based on these N noise values, N input noise values ​​are obtained. Figure 2 This will reduce the number of input noises to N. Figure 2 The images are fused with N clothing mask images to obtain N fused images 2. Using the first feature information of the preset clothing as the denoising control condition, the N fused images 2 are denoised using noise prediction model 1 to obtain N noise prediction values ​​2. Based on these N noise prediction values ​​2 and the aforementioned N noise values ​​2, the model loss 2 is determined, and the noise prediction model 1 is updated based on the loss 2 to obtain noise prediction model 2. Next, the electronic device generates N noise values ​​3, denoted as... Based on these N noise values, N input noise values ​​are obtained. Figure 3 This will reduce the number of input noises to N. Figure 3 The image is fused with N clothing mask images to obtain N fused images 3. Using the first feature information of the clothing as a preset denoising control condition, the N fused images 3 are denoised using noise prediction model 2 to obtain N noise prediction values ​​3. Based on these N noise prediction values ​​3 and the aforementioned N noise values ​​3, the model loss 3 is determined, and the noise prediction model 2 is updated based on the loss 3 to obtain noise prediction model 3. This process is repeated, and the electronic device generates N noise values ​​20, denoted as... Based on these N noise values, N input noise values ​​are obtained. Figure 20 This will reduce the number of input noises to N. Figure 20 The images are fused with N clothing mask images to obtain N fused images 20. Using the first feature information of the preset clothing as the denoising control condition, the N fused images 20 are denoised by the noise prediction model 19 to obtain N noise prediction values ​​20. Based on the N noise prediction values ​​20 and the aforementioned N noise values ​​20, the model loss 20 is determined, and the noise prediction model 19 is updated based on the loss 20 to obtain the noise prediction model 20.

[0225] Next, the electronic device selects new training samples from the training dataset and updates the parameters in the noise prediction model 20 in the same way as described above. This process is repeated multiple times until the training termination condition is met, resulting in the trained noise prediction model. The training termination condition includes either reaching a preset number of training iterations or the model's loss reaching a preset value.

[0226] The model training method provided in this application involves acquiring training samples, including training images and training video data. The training images include preset clothing, and the video objects in the training video data wear the preset clothing. Next, feature extraction is performed on the preset clothing in the training images to obtain first feature information of the preset clothing. The area to be dressed in each of the N image frames included in the training video data is masked to obtain N clothing mask images. These N clothing mask images are then fused with one of the N input noise images to obtain N fused images. Furthermore, the first feature information of the preset clothing serves as the control condition for the noise prediction model. The noise prediction model predicts noise in the N fused images to obtain N noise prediction values. Based on these N noise prediction values ​​and the N noise values ​​corresponding to the N input noise images, the loss of the noise prediction model is determined. Based on this loss, the noise prediction model is trained to obtain the trained noise prediction model. Therefore, in this embodiment, when the N fused images corresponding to the training video data are input into the noise prediction model as a time series, the noise prediction model not only focuses on the internal information of the video frames but also on the relevant information between the video frame sequences during the noise prediction process, making the final generated video data after the costume change more continuous. Furthermore, in the process of using the noise prediction model to predict noise in the N fused images, this embodiment considers not only the first feature information of the preset clothing but also the relevant information of the clothing mask image. The clothing mask image includes other information in the image frame besides the area to be changed, such as the pose of the video object and the environment in which the video object is located. This ensures that the final generated denoised image not only includes the preset clothing but also retains other original information from the image frame, thereby improving the effect of the video costume change.

[0227] The above describes the model training process; the following describes the video costume change process.

[0228] Figure 20 This is a flowchart illustrating a video data processing method provided in an embodiment of this application. The execution entity of this embodiment can be a device with video costume-changing functionality, such as a video data processing device. This video data processing device can be the one described above. Figure 1 The server 102 or terminal device 101 shown can also be Figure 1The system consists of server 102 or terminal device 101. For ease of description, the following embodiments use electronic devices as examples to illustrate the methods of this application.

[0229] like Figure 20 As shown, the video data processing method of this application embodiment includes:

[0230] S201. Obtain images including preset clothing and raw video data including video objects. The raw video data includes N image frames.

[0231] Where N is a positive integer.

[0232] It should be noted that the data used in the embodiments of this application and the process of obtaining this data comply with the relevant laws and regulations.

[0233] The purpose of this application embodiment is to replace the clothing of the video object in the original video data with preset clothing to obtain video data after the clothing change, in which the video object in the video data after the clothing change wears the preset clothing.

[0234] The video object in this application embodiment can be a real biological entity, such as a person or an animal. Optionally, the video object in this application embodiment can also be a virtual object, such as a virtual person or a virtual animal. This application embodiment does not limit the specific type of video object.

[0235] The image mentioned above that includes preset clothing can be any image that includes preset clothing. This application embodiment does not limit the type of preset clothing; for example, it can be clothing, accessories, props, etc.

[0236] The original video data in this application embodiment can be video data uploaded by an object, such as video data of a video object recorded for the object. Optionally, the original video data can also be video data generated by a computer. This application embodiment does not limit the specific type of original video data or the method of obtaining it.

[0237] The specific implementation process of S201 can be referred to the relevant description of S101 above, and will not be repeated here.

[0238] S202. Extract features from the preset clothing in the image, and obtain the first feature information of the preset clothing.

[0239] This application does not limit the specific method by which an electronic device extracts the first feature information of a preset garment from an image including the preset garment.

[0240] In some embodiments, the electronic device uses a clothing feature extraction model to extract features from the preset clothing in the image, thereby obtaining first feature information of the preset clothing.

[0241] This application does not impose any restrictions on the specific network structure of the clothing feature extraction model.

[0242] In some embodiments, the clothing feature extraction model of this application is as follows: Figure 5 As shown, this clothing feature extraction model is used to extract detailed features and spatial details of a preset garment from an image containing that garment, in order to obtain local feature information of the preset garment. This local feature information is recorded as the first feature information of the target image. It includes M second network modules ( Figure 5 The image shows 6 second network modules. The electronic device uses these M second network modules to extract the detailed and spatial features of the preset clothing in an image, including the preset clothing, and obtains the first feature information of the preset clothing corresponding to each of the M second network modules.

[0243] The specific implementation process of S202 can be referred to the relevant description of S102 above, and will not be repeated here.

[0244] S203. Mask the areas to be dressed in the N image frames to obtain N clothing mask images.

[0245] The area to be dressed up is at least a portion of the clothing area of ​​the video object in N image frames, where N is a positive integer.

[0246] It should be noted that the value of N here may differ from the value of N used in the model training process described above. Furthermore, the number of image frames included in different original video data may also vary; therefore, N here simply indicates the number of image frames included in the current original video data.

[0247] In some embodiments, if the original video data includes too many frames, in order to alleviate the effect deviation caused by the difference in the number of frames during training and use, this application embodiment uses a sliding smoothing method for inference. That is, for a video sequence of length P, first take N frames and execute the method of this application embodiment. After an interval of s (for example, s=4, but other values ​​are possible), continue to take the next N frames and continue to execute the method of this application embodiment. Repeat this process until all frames of the original video data have been traversed. The results are averaged based on the number of times each frame is inferred. The following description uses an example where the original video data includes N image frames.

[0248] The original video data in this embodiment includes N image frames. This original video data is then divided into a series of image frames, denoted as...

[0249] For each of these N image frames, the electronic device masks the area to be dressed in each image frame, thereby obtaining the clothing mask image for each image frame.

[0250] In the embodiments of this application, the specific methods by which the electronic device determines the clothing mask image of each of the N image frames are basically the same. For ease of description, we will use one image frame as an example, for example, the j-th image frame in the N image frames, where j is a positive integer less than or equal to N.

[0251] The embodiments of this application do not limit the specific method of masking the area to be dressed in the image frame to obtain the clothing mask image of the image frame.

[0252] In some embodiments, for the j-th image frame out of N image frames, keypoint detection is performed on the j-th image frame to obtain the keypoint pose information of the video object in the j-th image frame.

[0253] Next, the electronic device segments different regions of the video object in the j-th image frame to obtain the region segmentation map of the j-th image frame.

[0254] S204. Fuse the N clothing mask images with one of the N input noise images to obtain N fused images.

[0255] In this process, a clothing mask image and an input noise image correspond to each other to obtain a fused image.

[0256] In this embodiment, for each of the N clothing mask images, an input noise image is determined. This embodiment does not limit the specific method by which the electronic device determines the input noise image. In one example, the electronic device randomly generates N noise values, denoted as... These N noise values ​​are denoted as N input noise maps.

[0257] Next, the electronic device fuses the N clothing mask images with one of the N input noise images to obtain N fused images.

[0258] In one possible implementation, for each of the N clothing mask images, the corresponding input noise image is concatenated with the clothing mask image along the channel dimension. For example, ... With N input noise graphs By stitching them together along the channel dimension, N fused images are obtained.

[0259] The specific implementation process of S204 can be referred to the relevant description of S104 above, and will not be repeated here.

[0260] S205. Using the first feature information of the preset clothing as the control condition of the noise prediction model, the noise in the N fused images is predicted by the noise prediction model to obtain N noise prediction values.

[0261] The noise prediction model is a model trained on video data.

[0262] The embodiments of this application do not limit the specific type of noise prediction model.

[0263] In some embodiments, the noise prediction model described above is a noise prediction model trained using the model training method described above. The model training process is as described in the embodiments above and will not be repeated here.

[0264] In this embodiment, the N fused images corresponding to the original video data are processed as a time series using a noise prediction model. During the processing, not only is the information within each image frame learned, but the features at the same position in each image frame are also interactively learned to complete the information exchange between frames. This ensures that the final generated images after the costume change are more continuous, thus improving the effect of the video costume change.

[0265] In some embodiments, such as Figure 21 As shown, the electronic device directly inputs N fused images into the noise prediction model, so that the noise prediction model uses the first feature information of the preset clothing as the control condition to denoise the N fused images and obtain N noise prediction values.

[0266] In some embodiments, S205 above includes the following steps S205-A to S205-C:

[0267] S205-A: Transform N fused images from pixel space to latent vector space to obtain the feature information of N fused images;

[0268] S205-B: Based on the feature information of N fused images, determine the input information of the noise prediction model;

[0269] S205-C: Using the first feature information of the preset clothing as the control condition of the noise prediction model, the noise prediction model processes the input information to obtain N noise prediction values.

[0270] In this implementation, to improve image processing speed and reduce computational workload, the electronic device can transform N fused images from pixel space to latent vector space to obtain feature information of N fused images. The main purpose of this part is to represent the features of the fused images using high-level semantic features, which not only reduces the spatial dimension of the fused images, thereby improving training speed, but also facilitates better interaction with the control conditions of the noise prediction model.

[0271] In some embodiments, the electronic device uses an image feature encoder to convert N fused images from pixel space to latent vector space to obtain feature information of the N fused images.

[0272] Next, the electronic device determines the input information of the noise prediction model based on the feature information of the N fused images. This application embodiment does not limit the specific method by which the sub-device determines the input information of the noise prediction model based on the feature information of the N fused images.

[0273] In some embodiments, the electronic device can directly determine the feature information of N fused images as the input information of the noise prediction model.

[0274] In some embodiments, S205-B above includes the following steps S205-B1 to S205-B3:

[0275] S205-B1. Perform keypoint detection on each of the N image frames to obtain the keypoint pose information of the video object in each image frame.

[0276] S205-B2: Based on the pose information of the key points of the video object in each image frame, determine the skeleton feature information of the video object in each image frame;

[0277] S205-B3: Determine the input information based on the skeleton feature information of the video object in each image frame and the feature information of each fused image.

[0278] In this implementation, to further enhance the video dress-up effect, the pose information of the video object is also considered during the dress-up process. Specifically, the electronic device performs keypoint detection on each of the N image frames to obtain the keypoint pose information of the video object in each image frame. Based on the keypoint pose information of the video object in each image frame, the skeleton feature information of the video object in each image frame is determined. For example, an electronic device can collect the keypoint pose information of the video object from N image frames. The pose encoder performs pose encoding processing on the input to obtain the skeleton feature information of the video object in each of the N image frames.

[0279] In this embodiment of the application, after the electronic device determines the skeleton feature information of the video object in each of the N image frames, such as... Figure 12 As shown, the input information is determined based on the skeleton feature information of the video object in each of the N image frames and the feature information of each fused image.

[0280] In this embodiment, after determining the input information of the noise prediction model based on the above steps, the electronic device inputs the input information and the first feature information of the preset clothing into the noise prediction model, so that the noise prediction model processes the input information with the first feature information of the preset clothing as the control condition to obtain N noise prediction values.

[0281] In some embodiments, the electronic device uses the first feature information of the preset clothing as the denoising control condition, and performs multiple denoising processes on the input information through a noise prediction model to obtain N noise prediction values.

[0282] In some embodiments, the noise prediction model described above includes M first network modules, and S205-C includes the following steps S205-C1 to S205-C3:

[0283] S205-C1. For the i-th first network module among the M first network modules, obtain the i-1-th feature information corresponding to the N image frames output by the (i-1)-th first network module, where i is a positive integer from 1 to M. If i equals 1, then the (i-1)-th feature information is the input information.

[0284] S205-C2: Using the first feature information of the preset clothing as the control condition, the i-th first network module processes the (i-1)th feature information corresponding to N image frames to obtain the i-th feature information corresponding to N image frames output by the i-th first network module.

[0285] S205-C3: Based on the Mth feature information corresponding to the N image frames output by the Mth first network module, and based on the Mth feature information corresponding to the N image frames, obtain N noise prediction values.

[0286] In the embodiments of this application, such as Figure 13 As shown, the noise prediction model includes M first network modules. For each of the M network modules, such as the i-th first network module, the electronic device first obtains the feature information output by the previous network module of the i-th first network module, that is, the (i-1)-th network module. For ease of description, the feature information output by the (i-1)-th network module is recorded as the (i-1)-th feature information corresponding to N image frames.

[0287] In this embodiment of the application, after the electronic device obtains the (i-1)th feature information corresponding to N image frames, it inputs the (i-1)th feature information corresponding to the N image frames into the i-th first network module, so that the i-th first network module processes the (i-1)th feature information corresponding to the N image frames with the first feature information of the preset clothing as the control condition, and obtains the i-th feature information corresponding to the N image frames output by the i-th first network module.

[0288] In some embodiments, the electronic device extracts first feature information of a preset garment from an image including preset garments using a garment feature extraction model. For example... Figure 5 As shown, the clothing feature extraction model includes M second network modules, which are connected one-to-one with the M first network modules in the noise prediction model. In this case, the specific method for extracting features from the preset clothing in the image including the preset clothing in S202 to obtain the first feature information of the preset clothing is as follows: Through the M second network modules, features are extracted from the preset clothing in the image including the preset clothing to obtain the first feature information of the preset clothing corresponding to each of the M second network modules. In this case, S205-C2 may include the following steps S205-C21 and S205-C22:

[0289] S205-C21. Obtain the first feature information of the preset clothing corresponding to the i-th second network module in the clothing feature extraction model;

[0290] S205-C22. Using the i-th first feature information of the preset clothing as the control condition, the i-th first network module processes the (i-1)-th feature information corresponding to N image frames to obtain the i-th feature information corresponding to N image frames.

[0291] In this implementation, the clothing feature extraction model includes M second network modules, and the noise prediction model includes M first network modules. These M second network modules are connected one-to-one with the M first network modules. For example, taking the i-th first network module among the M first network modules as an example, this i-th first network module is connected to the i-th second network module. Therefore, the electronic device acquires the first feature information of the preset clothing corresponding to the i-th second network module. For ease of description, the first feature information of the preset clothing corresponding to the i-th second network module is denoted as the i-th first feature information of the preset clothing. Thus, as... Figure 14 As shown, the electronic device uses the i-th first feature information of the preset clothing as the control condition, and processes the (i-1)-th feature information corresponding to N image frames through the i-th first network module to obtain the i-th feature information corresponding to N image frames.

[0292] In some embodiments, the first network module of this application includes an attention layer, denoted as the first attention layer, and the second network module also includes an attention layer, denoted as the second attention layer. It should be noted that the first and second network modules may also include other network layers, such as convolutional layers, etc., and this application does not impose any limitations on this.

[0293] In one example, if the i-th first feature information of the preset clothing is the input information of the second attention layer in the i-th second network module, then the above S205-C22 includes the following steps S205-C221 to S205-C224:

[0294] S205-C221. Based on the (i-1)th feature information corresponding to N image frames, obtain the input information of the first attention layer included in the i-th first network module;

[0295] S205-C222: Fuse the input information of the first attention layer and the input information of the second attention layer to obtain fused input information;

[0296] S205-C223. Self-attention processing is performed on the fused input information corresponding to each image frame in the fused input information through the first attention layer, and the first attention processing results corresponding to the N image frames output by the i-th first network module are obtained.

[0297] S205-C224. Based on the first attention processing results corresponding to N image frames, determine the i-th feature information corresponding to the N image frames.

[0298] In the embodiments of this application, such as Figure 15 As shown, the electronic device uses the input information of the second attention layer of each of the M second network modules included in the clothing feature extraction model as the control condition of the corresponding first network module. For example, taking the i-th first network module as an example, the electronic device fuses the input information of the second attention layer in the i-th second network module as the control condition of the i-th first network module with the (i-1)-th feature information corresponding to N image frames, and uses this as the input information of the first attention layer in the i-th first network module.

[0299] Specifically, for the i-th first network module, the electronic device first obtains the input information of the first attention layer based on the (i-1)-th feature information corresponding to the above N image frames.

[0300] In some embodiments, the electronic device performs inter-frame three-dimensional convolution processing on the (i-1)th feature information corresponding to N image frames to obtain the convolution result, and then determines the input information of the first attention layer based on the convolution result.

[0301] In one possible implementation, such as Figure 17 As shown, the electronic device fuses the key K in the input information of the first attention layer with the key K in the input information of the second attention layer to obtain a new key K, and fuses the value V in the input information of the first attention layer with the value V in the input information of the second attention layer to obtain a new value V; the new key K, the query Q of the first attention layer and the new value V are determined as the fused input information.

[0302] Next, the electronic device inputs the aforementioned fused input information into the first attention layer of the i-th first network module. This first attention layer performs intra-frame attention processing on the fused input information corresponding to each image frame in the fused input information, obtaining the first attention processing results corresponding to N image frames. This application embodiment does not limit the specific type of the first attention layer. In one example, the first attention layer in this application embodiment is a self-attention layer. The electronic device performs self-attention processing on the fused input information corresponding to each image frame in the fused input information through the first attention layer, obtaining the first attention processing results corresponding to each of the N image frames. The first attention processing results corresponding to the N image frames are then concatenated to obtain the first attention processing result corresponding to the N image frames.

[0303] After determining the first attention processing results corresponding to N image frames based on the above steps, the electronic device executes the above steps S205-C224, that is, based on the first attention processing results corresponding to N image frames, it determines the i-th feature information corresponding to N image frames.

[0304] In one possible implementation, the electronic device performs inter-frame attention processing on the first attention processing results corresponding to N image frames to obtain the second attention processing results corresponding to N image frames; based on the first attention processing results and the second attention processing results corresponding to N image frames, the i-th feature information corresponding to N image frames is obtained.

[0305] In some embodiments, to further enhance the effect of video dress-up, the electronic device further extracts global features of the preset clothing from an image including the preset clothing to obtain second feature information of the preset clothing, and then... Figure 19 As shown, the first and second feature information of the preset clothing are used as the control conditions of the noise prediction model. The noise prediction model is used to predict the noise in N fused images to obtain N noise prediction values.

[0306] The specific implementation process of S205 can be referred to the relevant description of S105 above, and will not be repeated here.

[0307] After the electronic device obtains N noise prediction values ​​based on the above steps, it executes the following step S205.

[0308] S206. Based on N noise prediction values, denoise the N input noise maps respectively to obtain N denoised images. The denoised images are used to generate updated video data containing video objects.

[0309] In this embodiment, the electronic device determines N noise prediction values ​​based on the above steps, and then performs denoising processing on the N input noise maps based on these N noise prediction values ​​to obtain N denoised images. The region corresponding to the area to be dressed up in these N denoised images presents a preset garment. These N denoised images are used to obtain updated video data of the video object. This updated video data also becomes the dressed-up video data, in which the target object wears the preset garment.

[0310] In some embodiments of this application, the electronic device performs multiple denoising operations on N input noise maps to obtain the final video data after the costume change.

[0311] Specifically, the electronic device takes the N denoised images as the new N input noise images and returns to execute step S205, which uses the first feature information of the preset clothing as the control condition of the noise prediction model. The noise prediction model predicts the noise in the new N input noise images to obtain new N noise prediction values. The new N noise prediction values ​​are then used to denoise the new N input noise images. This process is repeated multiple times, for example, K times, to finally generate the video data after the costume change.

[0312] For example, suppose K equals 20. Figure 22 As shown, the electronic device first generates N noise values, denoted as... Based on these N noise values, N input noise values ​​are obtained. Figure 1 , to convert these N input noise Figure 1 The images are fused with N clothing mask images to obtain N fused images 1. In the first denoising process, the electronic device uses the preset first feature information of the clothing as a control condition and employs a noise prediction model to analyze the N input noise... Figure 1 The noise in the input is predicted, resulting in N noise prediction values ​​1. Based on these N noise prediction values ​​1, the N input noises are then analyzed. Figure 1First, denoising is performed to obtain N denoised images 1 after the first denoising. Next, the electronic device, using the preset first feature information of the clothing as a control condition, predicts the noise in the N denoised images 1 using a noise prediction model, obtaining N noise prediction values ​​2. Based on these N noise prediction values ​​2, denoising is performed on the N denoised images 1 again, resulting in N denoised images 2 after the second denoising. Then, the electronic device, using the preset first feature information of the clothing as a control condition, predicts the noise in the N denoised images 2 using a noise prediction model, obtaining N noise prediction values ​​3. Based on these N noise prediction values ​​3, denoising is performed on the N denoised images 2 again, resulting in N denoised images 3 after the third denoising.

[0313] This process is repeated. The electronic device uses the first feature information of the preset clothing as a control condition and predicts the noise in N denoised images 19 using a noise prediction model, obtaining N noise prediction values ​​20. Based on these N noise prediction values ​​20, the N denoised images 19 are denoised to obtain N denoised images 20 after the 20th denoising. The noise in these N denoised images 20 after the 20th denoising is removed, and the video object in the image is wearing the preset clothing. Therefore, the video data after the clothing change can be obtained based on these N denoised images 20 after the 20th denoising. For example, these N denoised images 20 after the 20th denoising can be identified as the N image frames included in the video data after the clothing change. Alternatively, image augmentation processing can be performed on these N denoised images 20 after the 20th denoising to obtain the video data after the clothing change.

[0314] The video data processing method provided in this application acquires an image including preset clothing and original video data including video objects, the original video data comprising N image frames. Next, feature extraction is performed on the preset clothing in the image including the preset clothing to obtain first feature information of the preset clothing. The area to be dressed in each of the N image frames is masked to obtain N clothing mask images, and each of the N clothing mask images is fused with one of the N input noise images to obtain N fused images. Then, using the first feature information of the preset clothing as the control condition of a noise prediction model, the noise in the N fused images is predicted by the noise prediction model to obtain N noise prediction values. Finally, based on the N noise prediction values, the N input noise images are denoised to obtain N denoised images, which are used to generate updated video data containing video objects. Therefore, it can be seen that when the N fused images corresponding to the original video data are input into the noise prediction model as a time series in this embodiment, the noise prediction model not only focuses on the internal information of the video frames but also on the relevant information between the video frame sequences during the noise prediction process, making the final generated video data after the costume change more continuous. In addition, in the process of using the noise prediction model to perform noise prediction on the N fused images, this embodiment considers not only the first feature information of the preset clothing but also the relevant information of the clothing mask image. The clothing mask image includes other information in the image frame besides the area to be changed, such as the pose of the video object and the environment in which the video object is located. This ensures that the final generated denoised image not only includes the preset clothing but also retains other original information in the image frame, thereby improving the effect of the video costume change.

[0315] The above text combined Figures 2 to 22 The method embodiments of this application are described in detail below, in conjunction with... Figure 23 The following describes in detail the device embodiments of this application.

[0316] Figure 23 This is a schematic block diagram of a video data processing apparatus provided in an embodiment of this application.

[0317] like Figure 23 As shown, the video data processing device 10 includes:

[0318] The acquisition unit 11 is used to acquire images including preset clothing and raw video data including video objects;

[0319] Extraction unit 12 is used to extract features from the preset clothing in the image including preset clothing to obtain the first feature information of the preset clothing;

[0320] The masking unit 13 is used to mask the clothing-to-be-changed areas in the N image frames included in the original video data to obtain N clothing mask images, wherein the clothing-to-be-changed areas are at least a portion of the clothing areas of the video object in the N image frames.

[0321] The fusion unit 14 is used to fuse the N clothing mask images with one of the N input noise images to obtain N fused images;

[0322] Prediction unit 15 is used to use the first feature information of the preset clothing as the control condition of the noise prediction model, and to predict the noise in the N fused images through the noise prediction model to obtain N noise prediction values.

[0323] The denoising unit 16 is used to perform denoising processing on the N input noise maps based on the N noise prediction values ​​to obtain N denoised images, and the N denoised images are used to generate updated video data containing the video object.

[0324] In some embodiments, the prediction unit 15 is specifically used to convert the N fused images from pixel space to latent vector space to obtain feature information of the N fused images; determine the input information of the noise prediction model based on the feature information of the N fused images; and use the first feature information of the preset clothing as the control condition of the noise prediction model to process the input information through the noise prediction model to obtain the N noise prediction values.

[0325] In some embodiments, the prediction unit 15 is specifically configured to perform keypoint detection on each of the N image frames to obtain keypoint pose information of the video object in each image frame; determine skeleton feature information of the video object in each image frame based on the keypoint pose information of the video object in each image frame; and determine the input information based on the skeleton feature information of the video object in each image frame and the feature information of each fused image.

[0326] In some embodiments, the prediction unit 15 is specifically used to, for each image frame, add the skeleton feature information of the video object in the image frame and the feature information of the fused image corresponding to the image frame to obtain the model input information of the image frame; and determine the model input information of the N image frames as the input information.

[0327] In some embodiments, the noise prediction model includes M first network modules, where M is a positive integer greater than 1. The prediction unit 15 is specifically used to obtain, for the i-th first network module among the M first network modules, the i-1th feature information corresponding to the N image frames output by the (i-1)-th first network module, where i is a positive integer from 1 to M. If i equals 1, then the (i-1)-th feature information is the input information. Using the first feature information of the preset clothing as a control condition, the i-th first network module processes the (i-1)-th feature information corresponding to the N image frames to obtain the i-th feature information corresponding to the N image frames output by the i-th first network module. This process is repeated to obtain the M-th feature information corresponding to the N image frames output by the M-th first network module. Based on the M-th feature information corresponding to the N image frames, the N noise prediction values ​​are obtained.

[0328] In some embodiments, the first feature information of the preset clothing is extracted by a clothing feature extraction model, which includes M second network modules. The M second network modules are connected one-to-one with the M first network modules. The extraction unit 12 is specifically used to extract the detailed features and spatial features of the preset clothing in the image including the preset clothing through the M second network modules to obtain the first feature information of the preset clothing corresponding to each of the M second network modules. The prediction unit 15 is specifically used to obtain the i-th first feature information of the preset clothing corresponding to the i-th second network module. Using the i-th first feature information of the preset clothing as a denoising control condition, the i-th first network module processes the (i-1)-th feature information corresponding to the N image frames to obtain the i-th feature information corresponding to the N image frames.

[0329] In some embodiments, the i-th first network module includes a first attention layer, the i-th second network module includes a second attention layer, and the i-th first feature information of the preset clothing is the input information of the second attention layer included in the i-th second network module; the prediction unit 15 is specifically used to obtain the input information of the first attention layer based on the (i-1)-th feature information corresponding to the N image frames; fuse the input information of the first attention layer and the input information of the second attention layer to obtain fused input information; perform intra-frame attention processing on the fused input information corresponding to each image frame in the fused input information through the first attention layer to obtain the first attention processing result corresponding to the N image frames output by the i-th first network module; and determine the i-th feature information corresponding to the N image frames based on the first attention processing result corresponding to the N image frames.

[0330] In some embodiments, the prediction unit 15 is specifically used to perform inter-frame three-dimensional convolution processing on the (i-1)th feature information corresponding to the N image frames to obtain the convolution result; and to determine the input information of the first attention layer based on the convolution result.

[0331] In some embodiments, the prediction unit 15 is specifically configured to fuse the key in the input information of the first attention layer with the key in the input information of the second attention layer to obtain a new key, and to fuse the value in the input information of the first attention layer with the value in the input information of the second attention layer to obtain a new value; and to determine the new key, the query of the first attention layer and the new value as the fused input information.

[0332] In some embodiments, the prediction unit 15 is specifically used to perform inter-frame cross-attention processing on the first attention processing results corresponding to the N image frames to obtain the second attention processing results corresponding to the N image frames; and based on the first attention processing results and the second attention processing results corresponding to the N image frames, to obtain the i-th feature information corresponding to the N image frames.

[0333] In some embodiments, the prediction unit 15 is further configured to extract the second feature information of the preset clothing in the image including the preset clothing; and to perform denoising processing on the N fused images by the noise prediction model using the first feature information and the second feature information of the preset clothing as denoising control conditions to obtain the N noise prediction values.

[0334] In some embodiments, the prediction unit 15 is further configured to extract global features of the preset clothing in the image including the preset clothing to obtain the second feature information of the preset clothing; and use the first feature information and the second feature information of the preset clothing as the control conditions of the noise prediction model to predict the noise in the N fused images through the noise prediction model to obtain the N noise prediction values.

[0335] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 23 The apparatus shown can perform the above-described video data processing method embodiments, and the foregoing and other operations and / or functions of each module in the apparatus are for implementing the above-described method embodiments, which will not be described in detail here for the sake of brevity.

[0336] Figure 24 This is a schematic block diagram of a model training device provided in an embodiment of this application.

[0337] like Figure 24As shown, the model training device 20 includes:

[0338] Acquisition unit 21 is used to acquire training samples, the training samples include training images and training video data, the training images are images including preset clothing, the video objects in the training video data are wearing the preset clothing, and the training video data includes N image frames;

[0339] Extraction unit 22 is used to extract features from the preset clothing in the training image to obtain the first feature information of the preset clothing;

[0340] Masking unit 23 is used to mask the clothing area to be changed in N image frames respectively to obtain N clothing mask images, wherein the clothing area to be changed is at least part of the clothing area of ​​the video object in the N image frames;

[0341] The fusion unit 24 is used to fuse the N clothing mask images with one of the N input noise images to obtain N fused images;

[0342] Prediction unit 25 is used to use the first feature information of the preset clothing as the control condition of the noise prediction model, and to predict the noise in the N fused images through the noise prediction model to obtain N noise prediction values.

[0343] Training unit 26 is used to determine the loss of the noise prediction model based on the N preset noise values ​​and the noise values ​​corresponding to the N input noise maps, and to train the noise prediction model based on the loss to obtain the trained noise prediction model.

[0344] In some embodiments, the prediction unit 25 is specifically used to convert the N fused images from pixel space to latent vector space to obtain feature information of the N fused images; determine the input information of the noise prediction model based on the feature information of the N fused images; use the first feature information of the preset clothing as the control condition of the noise prediction model, and process the input information through the noise prediction model to obtain the N noise prediction values.

[0345] In some embodiments, the prediction unit 25 is specifically used to perform keypoint detection on each of the N image frames to obtain keypoint pose information of the video object in each image frame; to encode the keypoint pose information of the video object in each image frame using a pose coding model to obtain skeleton feature information of the video object in each image frame; and to determine the input information based on the skeleton feature information of the video object in each image frame and the feature information of each fused image. The training unit 26 is specifically used to synchronously train the noise prediction model and the pose coding model based on the loss to obtain the trained noise prediction model and the trained pose coding model.

[0346] In some embodiments, the extraction unit 22 is specifically used to extract features from the preset clothing in the training image using the clothing feature extraction model to obtain the first feature information of the preset clothing; the training unit 26 is specifically used to synchronously train the noise prediction model and the clothing feature extraction model based on the loss to obtain the trained noise prediction model and the trained clothing feature extraction model.

[0347] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 24 The apparatus shown can execute the embodiments of the above-described model training method, and the foregoing and other operations and / or functions of each module in the apparatus are for implementing the above-described method embodiments, which will not be described in detail here for the sake of brevity.

[0348] The apparatus of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0349] Figure 25 This is a schematic block diagram of an electronic device provided in an embodiment of this application. The electronic device may be the terminal device or server described above.

[0350] like Figure 25 As shown, the electronic device 40 may include:

[0351] The system includes a memory 41 and a processor 42. The memory 41 stores a computer program 43 and transfers the computer program 43 to the processor 42. In other words, the processor 42 can retrieve and run the computer program 43 from the memory 41 to implement the methods described in the embodiments of this application.

[0352] For example, the processor 42 can be used to execute the steps in the above method according to the instructions in the computer program 43.

[0353] In some embodiments of this application, the processor 42 may include, but is not limited to:

[0354] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0355] In some embodiments of this application, the memory 41 includes, but is not limited to:

[0356] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0357] In some embodiments of this application, the computer program 43 may be divided into one or more modules, which are stored in the memory 41 and executed by the processor 42 to complete the page recording method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 43 in the electronic device.

[0358] like Figure 25 As shown, the electronic device 40 may further include:

[0359] Transceiver 44, which can be connected to processor 42 or memory 41.

[0360] The processor 42 can control the transceiver 44 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 44 may include a transmitter and a receiver. The transceiver 44 may further include antennas, and the number of antennas may be one or more.

[0361] It should be understood that the various components in the electronic device 40 are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0362] According to one aspect of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0363] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in the above-described method embodiments.

[0364] In other words, when implemented using software, it can be implemented wholly or partially in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0365] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0366] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0367] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0368] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video data processing method, characterized in that, include: Acquire an image containing a preset outfit, and raw video data containing a video object, wherein the raw video data includes N image frames, where N is a positive integer; Feature extraction is performed on the preset clothing in the image to obtain first feature information; The areas to be dressed up in the N image frames are masked respectively to obtain N clothing mask images, wherein the areas to be dressed up are at least a portion of the clothing areas of the video object in the N image frames; The N clothing mask images are fused with one of the N input noise images to obtain N fused images; Using the first feature information of the preset clothing as the control condition of the noise prediction model, the noise in the N fused images is predicted by the noise prediction model to obtain N noise prediction values; The N input noise maps are denoised based on the N noise prediction values ​​to obtain N denoised images, which are used to generate updated video data containing the video object.

2. The method according to claim 1, characterized in that, The first feature information of the preset clothing is used as the control condition for the noise prediction model. The noise prediction model is used to predict the noise in the N fused images to obtain N noise prediction values, including: The N fused images are transformed from pixel space to latent vector space to obtain the feature information of the N fused images; Based on the feature information of the N fused images, the input information of the noise prediction model is determined; Using the first feature information of the preset clothing as the control condition of the noise prediction model, the noise prediction model processes the input information to obtain the N noise prediction values.

3. The method according to claim 2, characterized in that, The step of determining the input information of the noise prediction model based on the feature information of the N fused images includes: Perform key point detection on each of the N image frames to obtain the key point pose information of the video object in each image frame; Based on the pose information of the key points of the video object in each image frame, determine the skeleton feature information of the video object in each image frame; The input information is determined based on the skeleton feature information of the video object in each image frame and the feature information of each fused image.

4. The method according to claim 3, characterized in that, The input information is determined based on the skeleton feature information of the video object in each image frame and the feature information of each fused image, including: For each image frame, the skeleton feature information of the video object in the image frame and the feature information of the fused image corresponding to the image frame are added together to obtain the model input information of the image frame; The model input information of the N image frames is determined as the input information.

5. The method according to any one of claims 2-4, characterized in that, The noise prediction model includes M first network modules, where M is a positive integer greater than 1. The noise control condition of the denoising prediction model is based on the first feature information of the preset clothing. The noise prediction model processes the input information to obtain the N noise prediction values, including: For the i-th first network module among the M first network modules, obtain the i-1-th feature information corresponding to the N image frames output by the (i-1)-th first network module, where i is a positive integer from 1 to M. If i equals 1, then the (i-1)-th feature information is the input information. Using the first feature information of the preset clothing as the control condition, the i-th first network module processes the (i-1)-th feature information corresponding to the N image frames to obtain the i-th feature information corresponding to the N image frames output by the i-th first network module; Based on the Mth feature information corresponding to the N image frames output by the Mth first network module, the N noise prediction values ​​are obtained.

6. The method according to claim 5, characterized in that, The first feature information of the preset clothing is obtained by extracting the clothing feature extraction model. The clothing feature extraction model includes M second network modules, and the M second network modules correspond one-to-one with the M first network modules. The step of extracting features from the preset clothing in the image including the preset clothing to obtain the first feature information of the preset clothing includes: extracting the detailed features and spatial features of the preset clothing in the image including the preset clothing through the M second network modules to obtain the first feature information of the preset clothing corresponding to each of the M second network modules; The step of using the first feature information of the preset clothing as the control condition, and processing the (i-1)th feature information corresponding to the N image frames through the i-th first network module to obtain the i-th feature information corresponding to the N image frames output by the i-th first network module includes: obtaining the i-th first feature information of the preset clothing corresponding to the i-th second network module; using the i-th first feature information of the preset clothing as the control condition, and processing the (i-1)th feature information corresponding to the N image frames through the i-th first network module to obtain the i-th feature information corresponding to the N image frames.

7. The method according to claim 6, characterized in that, The i-th first network module includes a first attention layer, the i-th second network module includes a second attention layer, and the i-th first feature information of the preset clothing is the input information of the second attention layer included in the i-th second network module; The step of using the i-th first feature information of the preset clothing as the control condition, and processing the (i-1)-th feature information corresponding to the N image frames through the i-th first network module to obtain the i-th feature information corresponding to the N image frames includes: Based on the (i-1)th feature information corresponding to the N image frames, the input information of the first attention layer included in the i-th first network module is obtained; The input information of the first attention layer and the input information of the second attention layer are fused to obtain fused input information; The first attention layer performs self-attention processing on the fused input information corresponding to each image frame in the fused input information to obtain the first attention processing result corresponding to the N image frames output by the i-th first network module; Based on the first attention processing results corresponding to the N image frames, the i-th feature information corresponding to the N image frames is determined.

8. The method according to claim 7, characterized in that, The step of obtaining the input information of the first attention layer based on the (i-1)th feature information corresponding to the N image frames includes: Perform inter-frame 3D convolution processing on the (i-1)th feature information corresponding to the N image frames to obtain the convolution result; Based on the convolution result, the input information of the first attention layer is determined.

9. The method according to claim 8, characterized in that, The step of fusing the input information of the first attention layer and the input information of the second attention layer to obtain fused input information includes: The keys in the input information of the first attention layer are fused with the keys in the input information of the second attention layer to obtain a new key, and the values ​​in the input information of the first attention layer are fused with the values ​​in the input information of the second attention layer to obtain a new value; The new key, the query from the first attention layer, and the new value are determined as the fused input information.

10. The method according to any one of claims 7 to 9, characterized in that, The step of determining the i-th feature information corresponding to the N image frames based on the first attention processing results includes: The first attention processing results corresponding to the N image frames are subjected to inter-frame cross-attention processing to obtain the second attention processing results corresponding to the N image frames; Based on the first attention processing result and the second attention processing result corresponding to the N image frames, the i-th feature information corresponding to the N image frames is obtained.

11. The method according to any one of claims 1-10, characterized in that, The method further includes: Global features of the preset clothing in the image are extracted to obtain the second feature information of the preset clothing; The noise prediction model uses the first feature information of the preset clothing as the control condition, and predicts the noise in the N fused images through the noise prediction model to obtain N noise prediction values, including: Using the first and second feature information of the preset clothing as the control conditions of the noise prediction model, the noise in the N fused images is predicted by the noise prediction model to obtain the N noise prediction values.

12. The method according to any one of claims 1-11, characterized in that, The process involves masking the areas to be dressed up in the N image frames to obtain N clothing mask images, including: For the j-th image frame among the N image frames, key point detection is performed on the j-th image frame to obtain the key point pose information of the video object in the j-th image frame, where j is a positive integer less than or equal to N; The video object in the j-th image frame is segmented to obtain the region segmentation map of the j-th image frame; Based on the key point pose information of the video object in the j-th image frame and the region segmentation map of the j-th image frame, the area to be dressed in the j-th image frame is masked to obtain the clothing mask map corresponding to the j-th image frame.

13. A model training method, characterized in that, include: Obtain training samples, which include training images and training video data. The training images are images including preset clothing, and the video objects in the training video data are wearing the preset clothing. The training video data includes N image frames, where N is a positive integer. Feature extraction is performed on the preset clothing in the training images to obtain the first feature information of the preset clothing; The regions to be dressed up in the N image frames included in the training video data are masked to obtain N clothing mask images; The N clothing mask images are fused with one of the N input noise images to obtain N fused images; Using the first feature information of the preset clothing as the control condition of the noise prediction model, the noise in the N fused images is predicted by the noise prediction model to obtain N noise prediction values; Based on the N preset noise values ​​and the noise values ​​corresponding to the N input noise maps, the loss of the noise prediction model is determined, and the noise prediction model is trained based on the loss to obtain the trained noise prediction model.

14. The method according to claim 13, characterized in that, The first feature information of the preset clothing is used as the control condition for the noise prediction model. The noise prediction model is used to predict the noise in the N fused images to obtain N noise prediction values, including: The N fused images are transformed from pixel space to latent vector space to obtain the feature information of the N fused images; Based on the feature information of the N fused images, the input information of the noise prediction model is determined; Using the first feature information of the preset clothing as the control condition of the noise prediction model, the noise prediction model processes the input information to obtain the N noise prediction values.

15. The method according to claim 13 or 14, characterized in that, The step of determining the input information of the noise prediction model based on the feature information of the N fused images includes: Perform key point detection on each of the N image frames to obtain the key point pose information of the video object in each image frame; The pose information of the key points of the video object in each image frame is encoded using a pose coding model to obtain the skeleton feature information of the video object in each image frame. The input information is determined based on the skeleton feature information of the video object in each image frame and the feature information of each fused image; The step of training the noise prediction model based on the loss to obtain the trained noise prediction model includes: The noise prediction model and the pose coding model are trained synchronously based on the loss to obtain the trained noise prediction model and the trained pose coding model.

16. The method according to any one of claims 13-15, characterized in that, The step of extracting features from the preset clothing in the training images to obtain the first feature information of the preset clothing includes: By using a clothing feature extraction model, features are extracted from the preset clothing in the training images to obtain the first feature information of the preset clothing; The step of training the noise prediction model based on the loss to obtain the trained noise prediction model includes: The noise prediction model and the clothing feature extraction model are trained synchronously based on the loss to obtain the trained noise prediction model and the trained clothing feature extraction model.

17. A video data processing apparatus, characterized in that, include: The acquisition unit is used to acquire images including preset clothing and raw video data of video objects, wherein the raw video data includes N image frames, where N is a positive integer; An extraction unit is used to extract features from the image including the preset clothing to obtain the first feature information of the preset clothing. A masking unit is used to mask the areas to be dressed up in the N image frames respectively to obtain N clothing mask images, wherein the areas to be dressed up are at least a portion of the clothing areas of the video object in the N image frames; A fusion unit is used to fuse the N clothing mask images with one of the N input noise images to obtain N fused images; The prediction unit is used to use the first feature information of the preset clothing as the control condition of the noise prediction model, and to predict the noise in the N fused images through the noise prediction model to obtain N noise prediction values. The denoising unit is used to denoise the N input noise maps based on the N noise prediction values ​​to obtain N denoised images. The area corresponding to the area to be dressed in the N denoised images presents the preset clothing, and is used to obtain the video data of the video object after dressing.

18. A model training device, characterized in that, include: An acquisition unit is used to acquire training samples, which include training images and training video data. The training images are images including preset clothing, and the video objects in the training video data are wearing the preset clothing. The training video data includes N image frames, where N is a positive integer. An extraction unit is used to extract features from a preset garment in the training image to obtain the first feature information of the preset garment. A masking unit is used to mask the areas to be dressed up in the N image frames respectively to obtain N clothing mask images, wherein the areas to be dressed up are at least a portion of the clothing areas of the video object in the N image frames; A fusion unit is used to fuse the N clothing mask images with one of the N input noise images to obtain N fused images; The prediction unit is used to use the first feature information of the preset clothing as the control condition of the noise prediction model, and to predict the noise in the N fused images through the noise prediction model to obtain N noise prediction values. The training unit is used to determine the loss of the noise prediction model based on the N preset noise values ​​and the noise values ​​corresponding to the N input noise maps, and to train the noise prediction model based on the loss to obtain the trained noise prediction model.

19. An electronic device, characterized in that, Including processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the method as described in any one of claims 1 to 12 or 13 to 16.

20. A computer-readable storage medium, characterized in that, Used to store computer programs; The computer program causes the computer to perform the method as described in any one of claims 1 to 12 or 13 to 16.

Images