Method and system for replacing object in video

Abstract

The invention discloses a method and system for replacing an object in a video, and relates to the technical field of video image processing, and the method comprises the following steps: obtaining video image data, and carrying out the preprocessing of the video image data; extracting video frame features, object replacement image features and text features according to the preprocessed video image data; and replacing the object in the video according to the extracted video frame features, image features and text features. According to the method and the system provided by the invention, the target object can be effectively and accurately positioned and tracked in the process of replacing the video object, and the natural sense and the continuity of the replaced object are kept, so that the coordination of the object and a video scene in the process of replacing the object is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video image processing, and in particular to a method and system for replacing objects in a video. Background Art

[0002] With the rapid development of artificial intelligence technology, especially breakthroughs in the field of video generation and editing technology, the application of video object replacement technology in the e-commerce field has become increasingly feasible.

[0003] At present, the core advantages of video object replacement technology are: the ability to efficiently reuse existing product video resources, and to achieve cost savings and improved marketing efficiency by replacing the products in the video with the merchant's own products. Video object replacement technology mainly relies on a powerful computing platform and complex post-production processes, which requires users to have certain technical knowledge and professional software operation capabilities.

[0004] However, the existing video object replacement technology cannot effectively realize real-time processing of video object replacement during the process of processing object replacement; and during the video shooting process, the object to be replaced may move in multiple positions, and even encounter occlusion during the movement, which makes it impossible to effectively and accurately locate and track the target object; in addition, the existing video object replacement technology cannot maintain the naturalness and continuity of the object after replacement when processing the transformation of the object in multiple positions and perspectives in the video; resulting in the inconsistency between the size of the object in the video and the replacement image during the video object replacement process, which leads to the problem of incoordination between the object and the video scene in the video output after the object is replaced. Summary of the invention

[0005] In order to overcome the problem that the existing video object replacement technology is unable to effectively and accurately locate and track the target object, and is unable to maintain the naturalness and continuity of the object after replacement, resulting in the object and the video scene being inconsistent in the video output after the object is replaced, the present invention proposes a method and system for object replacement in a video, which can effectively and accurately locate and track the target object, and maintain the naturalness and continuity of the object after replacement, thereby improving the coordination between the object and the video scene during the object replacement process.

[0006] To achieve the purpose of the present invention, the present invention adopts the following technical solutions: A method for replacing an object in a video, the method comprising the following steps: Acquire video image data, and preprocess the video image data; Extracting video frame features, object replacement image features and text features based on the preprocessed video image data; Objects in the video are replaced based on the extracted video frame features, image features and text features.

[0007] In the above technical scheme, preprocessing of the acquired video image data can improve the practicality and reliability of the data; the extracted video frame features, object replacement image features and text features can intuitively reflect the positioning and tracking information of the target object from the data, and then replace the objects in the video according to the correlation between different features, and maintain the naturalness and coherence of the objects after replacement, thereby improving the coordination between the objects and the video scenes during the object replacement process.

[0008] Furthermore, the process of preprocessing the video image data includes: De-noising and framing the video data in the video image data to extract a number of de-noised video frames; The object replacement image data in the video image data is denoised and normalized, and a text description is added to the object replacement image data.

[0009] In the above technical scheme, denoising the video data can reduce the complexity of the data processing process and improve the efficiency of data processing. Frame processing of the video data can better locate and track the target object to improve the accuracy of locating and tracking the target object. Denoising and normalization of image data can reduce the complexity of the data and remove noise in the image while maintaining the image details. Adding text description to the object replacement image data can improve the feature representation capability of the image data, thereby improving the practicality and reliability of the image data.

[0010] Furthermore, the process of extracting video frame features, image features and text features includes: The preprocessed video image data is divided into a training set and a validation set, and a video object replacement model is constructed, wherein the video object replacement model includes a video frame feature module, an object replacement image feature module, a text feature module and an object replacement module; Input the data in the training set into the constructed object replacement model in the video, and set up several rounds of iterative optimization training to optimize the parameters of the object replacement model in the video; Use the video frame feature module to extract the video frame features in the training set; The object replacement image feature module is used to extract the object replacement image features in the training set; Use the text feature module to extract text features in the training set; The object replacement module replaces the objects in the video according to the extracted video frame features, object replacement image features and text features, and outputs the object replacement results; The object replacement results after each round of training are verified using the validation set, and a loss function is set to optimize the parameters of the object replacement model in the video after each round of training. When the preset iterative optimization training round ends or the loss function converges, the trained object replacement model in the video is obtained. The trained object replacement model in the video is used to replace the video image data to be replaced, and the object replacement result is output.

[0011] In the above technical solution, the constructed object replacement model in the video can, after training and verification, efficiently recognize and analyze the input video image data, so as to effectively and accurately locate and track the target object, and maintain the naturalness and continuity of the object after replacement, thereby improving the coordination between the object and the video scene during the object replacement process.

[0012] Furthermore, the process of extracting video frame features in the training set using the video frame feature module includes: Use VAE encoder to extract video frame features from each video frame; An image segmentation method is used to perform image segmentation on each video frame, and a foreground mask and a background mask corresponding to each video frame are extracted, and a corresponding foreground attenuation mask and a background attenuation mask are extracted according to the extracted foreground mask and background mask; Extracting the noisy video frame features according to the video frame features, the foreground attenuation mask and the background attenuation mask; Among them, the foreground mask represents the object to be replaced, and the background mask represents the background of the image in the video frame.

[0013] Furthermore, the process of extracting the corresponding foreground attenuation mask and background attenuation mask includes: The distance transform algorithm is used to calculate the distance from each pixel in the image to the nearest foreground pixel, and the corresponding distance image is generated according to the distance transform result; Set the attenuation factor and determine the maximum and minimum values ​​of the attenuation factor; According to the distance image, the maximum and minimum values ​​of the attenuation factor, a linear attenuation function is used to map the distance to the value of the attenuation factor. The expression is: ; A foreground attenuation mask and a background attenuation mask are extracted according to the value of the attenuation factor; Among them, d represents the distance transformation result, Indicates the maximum distance value in the distance transform.

[0014] Furthermore, the process of extracting features of noisy video frames includes: Set the noise level sequence ; Based on the noise level sequence, the foreground attenuation mask and the background attenuation mask are denoised using element-by-element multiplication, and the expression is: ; The noise and the foreground attenuation mask and background attenuation mask after noise addition are fused into the video frame features to obtain the noisy video frame features, which are expressed as: ; Where T is the total number of steps in the noise addition process, Indicates no noise, is the largest noise category, represents the noise used in the Gaussian distribution, represents element-wise multiplication, M represents the foreground attenuation mask and the background attenuation mask, and t represents the time step.

[0015] In the above technical scheme, by adopting the VAE encoder to extract video frame features from each video frame, the semantic information of the video frame can be effectively captured to convert the frame into a low-dimensional table of high-dimensional data; the foreground mask and background mask corresponding to each video frame are extracted, and the corresponding foreground attenuation mask and background attenuation mask are extracted based on the extracted foreground mask and background mask, which can maintain the integrity of the background elements in the process of replacing the target object, and maintain the naturalness and coherence of the object after replacement, thereby improving the coordination between the object and the video scene during the object replacement process.

[0016] Furthermore, the object replacement image feature module uses the clip image editor to replace image features from objects in the training set; The text feature module uses the clip text editor to extract text features from the training set.

[0017] In the above technical solution, the clip image editor is used to replace image features from objects in the training set, which can extract high-level semantic information of the image. Combined with the semantic information of the text features, it can reflect the correlation characteristics of the image data and the text data, thereby improving the naturalness and coherence of the object after replacement during the object replacement process, and improving the coordination between the object and the video scene.

[0018] Furthermore, the object replacement module replaces the object in the video according to the extracted video frame features, object replacement image features and text features, and the process includes: Inputting the noisy video frame features, the denoised historical video frame features, the object replacement image features and the text features into the object replacement module; According to the input noisy video frame features, denoised historical video frame features, object replacement image features and text features, the object replacement module uses the conditional diffusion method to iteratively denoise the video frame features and outputs the denoised video frame features. The expression is: ; According to the object replacement image features and text features, the objects in the denoised video frame features are replaced, and the VAE decoder is used to decode the video frame features after object replacement to output the video after object replacement.

[0019] in, represents the noisy video frame features at time step t, Represents the features of the denoised video frame.

[0020] In the above technical scheme, according to the input noisy video frame features, the denoised historical video frame features, the object replacement image features and the text features, the object replacement module uses the conditional diffusion method to iteratively denoise the video frame features, which can make the potential representation in the data closer and closer to the original clean data, while ensuring that the generated content conforms to the text description and picture features. Finally, according to the object replacement image features and text features, the objects in the denoised video frame features are replaced, and the VAE decoder is used to decode the video frame features after object replacement, and the video after object replacement is output, which can effectively locate and track the target object, and maintain the naturalness and coherence after object replacement, thereby improving the coordination between the object and the video scene during the object replacement process.

[0021] Furthermore, the process of setting the loss function to optimize the parameters of the object replacement model in the video after each round of training includes: Calculate the difference loss between video frames, the expression is: ; Calculate the optical flow consistency loss of consecutive frames in the video frame, the expression is: ; Calculate the video temporal coherence loss, the expression is: ; According to the video frame difference loss, optical flow consistency loss and video temporal coherence loss, the total loss of each round of optimization training of the object replacement model in the video is calculated, and the expression is: ; Optimize the parameters of the object replacement model in the video being trained according to the total loss of each round of optimization training of the object replacement model in the video; in, represents the real image, represents the image decoded by VAE, N represents the total number of frames, represents the L2 norm, T represents the total number of frames of the extracted video, represents the t-th frame image, represents the real optical flow from the tth frame to the t+2th frame, represents the optical flow of the video frames generated by the model from the tth frame to the t+2th frame, It represents the feature vector of the generated video after ResNet+LSTM encoding. Represents the feature vector of the real video after being encoded by ResNet+LSTM. represents the dot product of the feature vectors of the generated video and the real video, represents the modulus of the feature vectors of the generated video and the real video, , , Represents the weight of each loss term.

[0022] In the above technical scheme, by setting different loss functions and optimizing the parameters of the object replacement model in the video during the object replacement training process, the target object can be effectively and accurately located and tracked, and the naturalness and continuity of the object after replacement can be maintained, thereby improving the coordination between the object and the video scene during the object replacement process.

[0023] A system for replacing objects in a video, the system comprising: A data acquisition module, used for acquiring video image data; A data processing module, used for preprocessing the video image data; A feature extraction module, used to extract video frame features, object replacement image features and text features based on the preprocessed video image data; The object replacement module is used to replace objects in the video based on the extracted video frame features, object replacement image features and text features.

[0024] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a method and system for replacing objects in a video, which can improve the practicability and reliability of the data by preprocessing the acquired video image data; the extracted video frame features, object replacement image features and text features can intuitively reflect the positioning and tracking information of the target object from the data, and then replace the object in the video according to the correlation between different features, and maintain the naturalness and coherence of the object after replacement, thereby improving the coordination between the object and the video scene during the object replacement process. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1A flowchart of the steps of a method for replacing an object in a video provided by an embodiment of the present application; Figure 2 A schematic diagram of the structure of an object replacement model in a video provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a system for replacing objects in a video provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Preferred embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0028] Embodiment 1: A method for replacing objects in a video, see Figure 1 , the method comprises the following steps: Step S1: Acquire video image data and pre-process the video image data; Step S2: extracting video frame features, object replacement image features and text features based on the preprocessed video image data; Step S3: replacing objects in the video according to the extracted video frame features, image features and text features.

[0029] In step S1, the process of preprocessing the video image data includes: De-noising and framing the video data in the video image data to extract a number of de-noised video frames; The object replacement image data in the video image data is denoised and normalized, and a text description is added to the object replacement image data.

[0030] It can be understood that denoising video data can reduce the complexity of the data processing process and improve the efficiency of data processing. Frame processing of video data can better locate and track the target object to improve the accuracy of locating and tracking the target object. Denoising and normalization of image data can reduce the complexity of the data and remove noise in the image while maintaining image details. Adding text descriptions to object replacement image data can improve the feature representation capability of image data, thereby improving the practicality and reliability of image data.

[0031] In this embodiment, preprocessing of the acquired video image data can improve the practicality and reliability of the data; the extracted video frame features, object replacement image features and text features can intuitively reflect the positioning and tracking information of the target object from the data, and then replace the objects in the video according to the correlation between different features, and maintain the naturalness and continuity of the objects after replacement, thereby improving the coordination between the objects and the video scenes during the object replacement process.

[0032] Embodiment 2: This embodiment is based on the first embodiment, and further describes step S2-step S3, as follows: In step S2, see Figure 2 ,The process of extracting video frame features, image features and text features includes: Step S21: dividing the preprocessed video image data into a training set and a verification set, and constructing a video object replacement model, wherein the video object replacement model includes a video frame feature module, an object replacement image feature module, a text feature module, and an object replacement module; Step S22: input the data in the training set into the constructed video object replacement model, and set several rounds of iterative optimization training to optimize the parameters of the video object replacement model; Step S23: extracting video frame features in the training set using a video frame feature module; Step S24: extracting object replacement image features in the training set using the object replacement image feature module; Step S25: extracting text features in the training set using a text feature module; Step S26: the object replacement module replaces the object in the video according to the extracted video frame features, object replacement image features and text features, and outputs the object replacement result; Step S27: Use the verification set to verify the object replacement results after each round of training, and set the loss function to optimize the parameters of the object replacement model in the video after each round of training. When the preset iterative optimization training round ends or the loss function converges, the trained object replacement model in the video is obtained, and the trained object replacement model in the video is used to replace the video image data to be replaced, and the object replacement result is output.

[0033] In step S23, see Figure 2 ,The process of extracting video frame features in the training set using the video frame feature module includes: Use VAE encoder to extract video frame features from each video frame; Use an image segmentation method (Segment Anything Model 2) to segment each video frame, extract the foreground mask and background mask corresponding to each video frame, and extract the corresponding foreground attenuation mask and background attenuation mask based on the extracted foreground mask and background mask, where the area with a value close to 1 represents the foreground (target object), and the area with a value close to 0 represents the background; Extracting the noisy video frame features according to the video frame features, the foreground attenuation mask and the background attenuation mask; Among them, the foreground mask represents the object to be replaced, and the background mask represents the background of the image in the video frame.

[0034] Specifically, the process of extracting video frame features from each video frame using the VAE encoder includes: Each frame of the video is encoded separately (the video is composed of a series of continuous frames, each frame is an independent image), and the video frame image is converted into a latent space representation through the VAE encoder, which is recorded as the video frame feature, which is a vector of fixed dimension. This vector captures the semantic information of the video frame.

[0035] Specifically, the process of extracting the corresponding foreground attenuation mask and background attenuation mask includes: A distance transform algorithm (Euclidean distance transform or city block distance transform) is used to calculate the distance from each pixel in the image to the nearest foreground pixel, and a corresponding distance image is generated based on the distance transform result, in which the value of each pixel represents its distance to the nearest foreground pixel. Sets the attenuation factor and determines the maximum and minimum values ​​of the attenuation factor; in this case, the attenuation factor of the foreground (closest foreground pixels) is 1, while the attenuation factor of the background (furthest foreground pixels) is close to 0.

[0036] According to the distance image, the maximum and minimum values ​​of the attenuation factor, a linear attenuation function is used to map the distance to the value of the attenuation factor. The expression is: ; The foreground attenuation mask and the background attenuation mask are extracted according to the value of the attenuation factor; the attenuation factor of the foreground pixel is 1, the attenuation factor of the background pixel is close to 0, and the pixels between the two have an attenuation factor that decreases from 1 to 0 according to their distance to the foreground; Among them, d represents the distance transformation result, Indicates the maximum distance value in the distance transform.

[0037] Specifically, the process of extracting features of noisy video frames includes: Set the noise level sequence ; Based on the noise level sequence, the foreground attenuation mask and the background attenuation mask are denoised by element-by-element multiplication (each noise added to the video frame feature will be fused with the attenuated foreground and background masks, so that the noise added to the foreground area is the largest, while the noise in the background area decreases as the distance from the foreground increases). The expression is: ; The noise is fused with the foreground attenuation mask and the background attenuation mask after the noise addition process into the video frame features. For each step (from t to T), the current potential representation At the beginning, noise is added to the current potential representation to obtain the noisy video frame feature, which is expressed as: ; Where T is the total number of steps in the noise addition process, Indicates no noise, is the largest noise category, represents the noise used in the Gaussian distribution, represents element-wise multiplication, M represents the foreground attenuation mask and the background attenuation mask, and t represents the time step.

[0038] It can be understood that by using the VAE encoder to extract video frame features from each video frame, the semantic information of the video frame can be effectively captured to convert the frame into a low-dimensional table of high-dimensional data; extracting the foreground mask and background mask corresponding to each video frame, and extracting the corresponding foreground attenuation mask and background attenuation mask based on the extracted foreground mask and background mask, it is possible to gradually introduce noise to each noise-free original video frame feature, simulate the gradual transition from data distribution to Gaussian noise distribution, and introduce randomness and diversity in the generation process. The method of gradually adding noise enables the model to capture the complex structure of the data during the generation process while maintaining control over the generation process. And it is possible to maintain the integrity of the background elements in the process of replacing the target object, and at the same time, maintain the naturalness and coherence of the object after replacement, thereby improving the coordination between the object and the video scene during the object replacement process.

[0039] In step S24 and step S25, see Figure 2 ,The object replacement image feature module uses the clip image editor to replace image features from objects in the training set; The text feature module uses the clip text editor to extract text features from the training set.

[0040] It can be understood that using the clip image editor to replace image features from objects in the training set can extract high-level semantic information of the image. Combined with the semantic information of the text features, it can reflect the correlation characteristics of the image data and the text data, thereby improving the naturalness and coherence of the object after replacement, and improving the coordination between the object and the video scene.

[0041] In step S26, see Figure 2 The object replacement module replaces the object in the video according to the extracted video frame features, object replacement image features and text features, and the process includes: Inputting the noisy video frame features, the denoised historical video frame features, the object replacement image features and the text features into the object replacement module; According to the input noisy video frame features, denoised historical video frame features, object replacement image features and text features, the object replacement module uses the conditional diffusion method to iteratively denoise the video frame features and outputs the denoised video frame features. The expression is: ; According to the object replacement image features and text features, the objects in the denoised video frame features are replaced, and the VAE decoder is used to decode the video frame features after object replacement to output the video after object replacement.

[0042] in, represents the noisy video frame features at time step t, Represents the features of the denoised video frame.

[0043] Specifically, the noisy video frame features are gradually restored to the noise-free video frame features (i.e., the process of generating video frames after replacing the subject). Here, the Conditional Diffusion Model (CDMs) is used, and the core is a U-Net structure, which is a common neural network architecture for image and video generation. U-Net effectively combines contextual information and position information through its encoder-decoder structure.

[0044] During the denoising process, the features of all previous video frames, text features, and feature information of the image to be replaced are used as auxiliary information to assist in the denoising of the video frame. The details of the process are as follows: For each step t, the denoised historical video frame features, text features, and image features are combined with the current video frame features. As the input of the denoising model, the noise at each time step t is predicted by the denoising model , this process can be described as:

[0045] in, is the feature of the noisy video frame at time step t, The parameter is A denoising model that is used to predict the noise based on the noisy input and time step information. is the predicted noise, Represent text and image features respectively, The historical video frame output after denoising by the denoising model can be understood as the sum of all the video frame features before the current video frame timestamp, calculating the L2 norm of the added vector (i.e. the square root of the sum of the squares of the vector elements), and then dividing the elements of each dimension in the added vector by this norm to obtain the normalized historical video frame features. (The purpose of using this feature is to hope that the model can utilize the video frame features of the previous time step) The model then uses this predicted noise to update the video frame features to gradually restore clean data. is the denoised video frame feature:

[0046] The denoising process is iterated until all T steps of iteration are completed. Each step gradually reduces the noise, making the potential representation closer and closer to the original clean data, while ensuring that the generated content conforms to the text description and image features. After decoding is completed, the video frame features are decoded back to the video frame image through the VAE decoder.

[0047] It can be understood that according to the input noisy video frame features, the denoised historical video frame features, the object replacement image features and the text features, the object replacement module uses the conditional diffusion method to iteratively denoise the video frame features, which can make the potential representation in the data closer and closer to the original clean data, while ensuring that the generated content conforms to the text description and picture features. Finally, according to the object replacement image features and text features, the objects in the denoised video frame features are replaced, and the VAE decoder is used to decode the video frame features after object replacement, and the video after object replacement is output, which can effectively locate and track the target object, and maintain the naturalness and coherence after object replacement, thereby improving the coordination between the object and the video scene during the object replacement process.

[0048] In step S27, the process of setting the loss function to optimize the parameters of the object replacement model in the video after each round of training includes: Calculate the difference loss between video frames, prepare the video frame images decoded by VAE and the real video frame images at the corresponding time points, and ensure that they have the same size and format.

[0049] Calculate the reconstruction loss, which measures the pixel-level difference between each video frame generated by the model and the real video frame. The expression is: ; Calculate the optical flow consistency loss of consecutive frames in the video frame. This loss function uses optical flow to measure the motion consistency between consecutive frames. The expression is: ; Calculate the video temporal coherence loss, the expression is: ; Specifically, a pre-trained ResNet+LSTM video encoder is used, and the parameters of this model are kept unchanged. All video frames of the generated video and the real video are input into the model in chronological order; the temporal information and semantic information between the video frames are encoded through ResNet+LSTM, and they are converted into image vectors of fixed dimensions. These image vectors contain not only the static information of the frame, but also the temporal information between the frames. Then, these image vectors are input into the LSTM model to obtain feature vectors that can represent the entire video content. Finally, the cosine similarity of the two video feature vectors is calculated to measure the similarity between the generated video and the real video while maintaining the temporal consistency and coherence of the two videos.

[0050] According to the video frame difference loss, optical flow consistency loss and video temporal coherence loss, the total loss of each round of optimization training of the object replacement model in the video is calculated, and the expression is: ; Optimize the parameters of the object replacement model in the video being trained according to the total loss of each round of optimization training of the object replacement model in the video; in, represents the real image, represents the image decoded by VAE, N represents the total number of frames, represents the L2 norm, T represents the total number of frames of the extracted video, represents the t-th frame image, represents the real optical flow from the tth frame to the t+2th frame, represents the optical flow of the video frames generated by the model from the tth frame to the t+2th frame, It represents the feature vector of the generated video after ResNet+LSTM encoding. Represents the feature vector of the real video after being encoded by ResNet+LSTM. represents the dot product of the feature vectors of the generated video and the real video, represents the modulus of the feature vectors of the generated video and the real video, , , Represents the weight of each loss term, which can be adjusted according to the actual situation. Here it is set to 2:3:5, and the proportion of video temporal coherence loss is the largest. By optimizing the model parameters and minimizing the loss function through back propagation and gradient descent algorithms, it can ensure that the generated video is as close as possible to the real video on multiple levels, thereby improving the quality of video generation. The trained video object replacement model can be applied to product display videos based on text prompts to replace products in the video, and adjust the texture and style of objects in the video according to the description of the prompts.

[0051] It can be understood that by setting different loss functions and optimizing the parameters of the object replacement model in the video during the object replacement training process, the target object can be effectively positioned and tracked, and the naturalness and continuity of the object after replacement can be maintained, thereby improving the coordination between the object and the video scene during the object replacement process.

[0052] In this embodiment, after training and verification, the constructed object replacement model in the video can efficiently recognize and analyze the input video image data, thereby effectively and accurately positioning and tracking the target object, and maintaining the naturalness and continuity of the object after replacement, thereby improving the coordination between the object and the video scene during the object replacement process.

[0053] Embodiment three: A system for object replacement in videos, see Figure 3 , the system comprising: A data acquisition module, used for acquiring video image data; A data processing module, used for preprocessing the video image data; A feature extraction module, used to extract video frame features, object replacement image features and text features based on the preprocessed video image data; The object replacement module is used to replace objects in the video based on the extracted video frame features, object replacement image features and text features.

[0054] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for replacing an object in a video, characterized in that: The method comprises the following steps: Acquire video image data, and preprocess the video image data; Extracting video frame features, object replacement image features and text features based on the preprocessed video image data; Objects in the video are replaced based on the extracted video frame features, object replacement image features and text features.

2. The method for replacing an object in a video according to claim 1, characterized in that: The process of preprocessing the video image data includes: De-noising and framing the video data in the video image data to extract a number of de-noised video frames; The object replacement image data in the video image data is denoised and normalized, and a text description is added to the object replacement image data.

3. The method for replacing an object in a video according to claim 2, characterized in that: The process of extracting video frame features, image features, and text features includes: The preprocessed video image data is divided into a training set and a validation set, and a video object replacement model is constructed, wherein the video object replacement model includes a video frame feature module, an object replacement image feature module, a text feature module and an object replacement module; Input the data in the training set into the constructed object replacement model in the video, and set up several rounds of iterative optimization training to optimize the parameters of the object replacement model in the video; Use the video frame feature module to extract the video frame features in the training set; The object replacement image feature module is used to extract the object replacement image features in the training set; Use the text feature module to extract text features in the training set; The object replacement module replaces the objects in the video according to the extracted video frame features, object replacement image features and text features, and outputs the object replacement results; The object replacement results after each round of training are verified using the validation set, and a loss function is set to optimize the parameters of the object replacement model in the video after each round of training. When the preset iterative optimization training round ends or the loss function converges, the trained object replacement model in the video is obtained. The trained object replacement model in the video is used to replace the video image data to be replaced, and the object replacement result is output.

4. The method for replacing an object in a video according to claim 3, characterized in that: The process of extracting video frame features in the training set using the video frame feature module includes: Use VAE encoder to extract video frame features from each video frame; An image segmentation method is used to perform image segmentation on each video frame, and a foreground mask and a background mask corresponding to each video frame are extracted, and a corresponding foreground attenuation mask and a background attenuation mask are extracted according to the extracted foreground mask and background mask; Extracting the noisy video frame features according to the video frame features, the foreground attenuation mask and the background attenuation mask; Among them, the foreground mask represents the object to be replaced, and the background mask represents the background of the image in the video frame.

5. The method for replacing an object in a video according to claim 4, characterized in that: The process of extracting the corresponding foreground attenuation mask and background attenuation mask includes: The distance transform algorithm is used to calculate the distance from each pixel in the image to the nearest foreground pixel, and the corresponding distance image is generated according to the distance transform result; Set the attenuation factor and determine the maximum and minimum values ​​of the attenuation factor; According to the distance image, the maximum and minimum values ​​of the attenuation factor, a linear attenuation function is used to map the distance to the value of the attenuation factor. The expression is: ； A foreground attenuation mask and a background attenuation mask are extracted according to the value of the attenuation factor; Among them, d represents the distance transformation result, Indicates the maximum distance value in the distance transform.

6. The method for replacing an object in a video according to claim 4, characterized in that: The process of extracting features of noisy video frames includes: Set the noise level sequence ; Based on the noise level sequence, the foreground attenuation mask and the background attenuation mask are denoised using element-by-element multiplication, and the expression is: ； The noise and the foreground attenuation mask and background attenuation mask after noise addition are fused into the video frame features to obtain the noisy video frame features, which are expressed as: ； Where T is the total number of steps in the noise addition process, Indicates no noise, is the largest noise category, represents the noise used in the Gaussian distribution, represents element-wise multiplication, M represents the foreground attenuation mask and the background attenuation mask, and t represents the time step.

7. The method for replacing an object in a video according to claim 3, characterized in that: The object replacement image feature module uses the clip image editor to replace image features from objects in the training set; The text feature module uses the clip text editor to extract text features from the training set.

8. The method for replacing an object in a video according to any one of claims 3 to 7, characterized in that: The object replacement module replaces objects in the video according to the extracted video frame features, object replacement image features and text features. The process includes: Inputting the noisy video frame features, the denoised historical video frame features, the object replacement image features and the text features into the object replacement module; According to the input noisy video frame features, denoised historical video frame features, object replacement image features and text features, the object replacement module uses the conditional diffusion method to iteratively denoise the video frame features and outputs the denoised video frame features. The expression is: ； According to the object replacement image features and text features, the objects in the denoised video frame features are replaced, and the VAE decoder is used to decode the video frame features after object replacement to output the video after object replacement. in, represents the noisy video frame features at time step t, Represents the features of the denoised video frame.

9. The method for replacing an object in a video according to claim 3, characterized in that: The process of setting the loss function to optimize the parameters of the object replacement model in each round of training includes: Calculate the difference loss between video frames, the expression is: ； Calculate the optical flow consistency loss of consecutive frames in the video frame, the expression is: ； Calculate the video temporal coherence loss, the expression is: ； According to the video frame difference loss, optical flow consistency loss and video temporal coherence loss, the total loss of each round of optimization training of the object replacement model in the video is calculated, and the expression is: ； Optimize the parameters of the object replacement model in the video being trained according to the total loss of each round of optimization training of the object replacement model in the video; in, represents the real image, represents the image decoded by VAE, N represents the total number of frames, represents the L2 norm, T represents the total number of frames of the extracted video, represents the t-th frame image, represents the real optical flow from the tth frame to the t+2th frame, represents the optical flow of the video frames generated by the model from the tth frame to the t+2th frame, It represents the feature vector of the generated video after ResNet+LSTM encoding. Represents the feature vector of the real video after being encoded by ResNet+LSTM. represents the dot product of the feature vectors of the generated video and the real video, represents the modulus of the feature vectors of the generated video and the real video, , , Represents the weight of each loss term.

10. A system for replacing objects in a video, characterized in that: The system comprises: A data acquisition module, used for acquiring video image data; A data processing module, used for preprocessing the video image data; A feature extraction module, used to extract video frame features, object replacement image features and text features based on the preprocessed video image data; The object replacement module is used to replace objects in the video based on the extracted video frame features, object replacement image features and text features.

Images