Method and system for replacing objects in video

Abstract

The present invention discloses a method and system for replacing objects in a video, relating to the technical field of video image processing. The method comprises the following steps: acquiring video image data and preprocessing the video image data; extracting video frame features, object replacement image features, and text features based on the preprocessed video image data; and replacing objects in the video based on the extracted video frame features, image features, and text features. The method and system proposed by the present invention can effectively and accurately locate and track target objects during video object replacement, while maintaining a natural feel and coherence after the object replacement, thereby improving the coordination between the object and the video scene during the object replacement process.

Description

Technical Field

[0001] The present invention relates to the technical field of video image processing, and in particular to a method and system for replacing objects in a video. Background Art

[0002] With the rapid development of artificial intelligence technology, especially breakthroughs in video generation and editing technology, the application of video object replacement technology in the e-commerce field has become increasingly feasible.

[0003] Currently, the core advantages of video object replacement technology include the ability to efficiently reuse existing product video resources, replacing the products in the video with the merchant's own products, thereby reducing costs and improving marketing efficiency. Video object replacement technology relies heavily on powerful computing platforms and complex post-production processes, requiring users to possess certain technical knowledge and professional software skills.

[0004] However, the existing video object replacement technology cannot effectively realize real-time processing of video object replacement during the process of processing object replacement; and during the video shooting process, the object to be replaced may move in multiple positions, or even be blocked during the movement, making it impossible to effectively locate and track the target object; in addition, the existing video object replacement technology cannot maintain the naturalness and continuity of the object after replacement when processing the transformation of the object in multiple positions and perspectives in the video; resulting in the inconsistency between the size of the object in the video and the object in the replacement image during the video object replacement process, which in turn leads to the problem of incoordination between the object and the video scene in the video output after the object replacement. Summary of the Invention

[0005] In order to overcome the problems of existing video object replacement technology that are unable to effectively and accurately locate and track the target object, and unable to maintain the naturalness and continuity of the object after replacement, resulting in the object and the video scene being inconsistent in the video output after the object replacement, the present invention proposes a method and system for object replacement in video, which can effectively and accurately locate and track the target object, and maintain the naturalness and continuity of the object after replacement, thereby improving the coordination between the object and the video scene during the object replacement process.

[0006] To achieve the purpose of the present invention, the present invention adopts the following technical solutions:

[0007] A method for replacing an object in a video, the method comprising the following steps:

[0008] Acquiring video image data and preprocessing the video image data;

[0009] Extract video frame features, object replacement image features, and text features based on the preprocessed video image data;

[0010] Objects in the video are replaced based on the extracted video frame features, image features, and text features.

[0011] In the above technical solution, preprocessing of the acquired video image data can improve the practicality and reliability of the data; the extracted video frame features, object replacement image features and text features can intuitively reflect the positioning and tracking information of the target object from the data, and then replace the objects in the video according to the correlation between different features, and maintain the naturalness and coherence of the objects after replacement, thereby improving the coordination between the objects and the video scene during the object replacement process.

[0012] Furthermore, the process of preprocessing the video image data includes:

[0013] De-noising and framing the video data in the video image data to extract a number of de-noised video frames;

[0014] Denoising and normalization processing are performed on the object replacement image data in the video image data, and text description is added to the object replacement image data.

[0015] In the above technical solution, denoising the video data can reduce the complexity of the data processing process and improve the efficiency of data processing. Frame processing of the video data can better locate and track the target object to improve the accuracy of locating and tracking the target object. Denoising and normalizing the image data can reduce the complexity of the data and remove noise in the image while maintaining the image details. Adding text descriptions to the object replacement image data can improve the feature representation ability of the image data, thereby improving the practicality and reliability of the image data.

[0016] Furthermore, the process of extracting video frame features, image features, and text features includes:

[0017] The preprocessed video image data is divided into a training set and a validation set, and a video object replacement model is constructed, wherein the video object replacement model includes a video frame feature module, an object replacement image feature module, a text feature module and an object replacement module;

[0018] Input the data in the training set into the constructed object replacement model in the video, and set up several rounds of iterative optimization training to optimize the parameters of the object replacement model in the video;

[0019] Use the video frame feature module to extract video frame features in the training set;

[0020] Use the object replacement image feature module to extract the object replacement image features in the training set;

[0021] Use the text feature module to extract text features in the training set;

[0022] The object replacement module replaces objects in the video based on the extracted video frame features, object replacement image features, and text features, and outputs the object replacement results;

[0023] The object replacement results after each round of training are verified using the validation set, and a loss function is set to optimize the parameters of the object replacement model in the video after each round of training. When the preset iterative optimization training round ends or the loss function converges, the trained object replacement model in the video is obtained. The trained object replacement model in the video is used to replace the video image data to be replaced, and the object replacement result is output.

[0024] In the above technical solution, the constructed object replacement model in the video can, after training and verification, efficiently identify and analyze the input video image data, thereby effectively and accurately positioning and tracking the target object, and maintaining the naturalness and continuity of the object after replacement, thereby improving the coordination between the object and the video scene during the object replacement process.

[0025] Furthermore, the process of extracting video frame features in the training set using the video frame feature module includes:

[0026] Use VAE encoder to extract video frame features from each video frame;

[0027] Performing image segmentation on each video frame using an image segmentation method, extracting a foreground mask and a background mask corresponding to each video frame, and extracting a corresponding foreground attenuation mask and a background attenuation mask based on the extracted foreground mask and background mask;

[0028] Extracting the features of the noisy video frame according to the video frame features, the foreground attenuation mask and the background attenuation mask;

[0029] Among them, the foreground mask represents the object to be replaced, and the background mask represents the background of the image in the video frame.

[0030] Furthermore, the process of extracting the corresponding foreground attenuation mask and background attenuation mask includes:

[0031] The distance transform algorithm is used to calculate the distance between each pixel in the image and the nearest foreground pixel, and the corresponding distance image is generated based on the distance transform result;

[0032] Set the attenuation factor and determine the maximum and minimum values ​​of the attenuation factor;

[0033] According to the distance image, the maximum and minimum values ​​of the attenuation factor, a linear attenuation function is used to map the distance to the value of the attenuation factor. The expression is:

[0034]

[0035] The foreground attenuation mask and the background attenuation mask are extracted according to the value of the attenuation factor;

[0036] Among them, d represents the distance transformation result, D max Indicates the maximum distance value in the distance transformation.

[0037] Furthermore, the process of extracting features of noisy video frames includes:

[0038] Set the noise level sequence σ1, σ2, σ3, ...., σ T ;

[0039] Based on the noise level sequence, the foreground attenuation mask and the background attenuation mask are noised using element-by-element multiplication. The expression is:

[0040] ∈′ t =σ t ⊙M;

[0041] The noise and the foreground attenuation mask and background attenuation mask after noise processing are fused into the video frame features to obtain the noisy video frame features, which are expressed as:

[0042] Z′ t =Z t-1 +∈′ t ;

[0043] Where T is the total number of steps in the noise addition process, σ T =0 means no noise, σ1 is the largest noise category, ∈′ t Represents the noise used in the Gaussian distribution, ⊙ represents element-wise multiplication, M represents the foreground attenuation mask and the background attenuation mask, and t represents the time step.

[0044] In the above technical solution, by adopting the VAE encoder to extract video frame features from each video frame, the semantic information of the video frame can be effectively captured to convert the frame into a low-dimensional table of high-dimensional data; the foreground mask and background mask corresponding to each video frame are extracted, and based on the extracted foreground mask and background mask, the corresponding foreground attenuation mask and background attenuation mask are extracted, which can maintain the integrity of the background elements in the process of replacing the target object, and maintain the naturalness and coherence of the object after replacement, thereby improving the coordination between the object and the video scene during the object replacement process.

[0045] Furthermore, the object replacement image feature module uses the clip image editor to replace image features from objects in the training set;

[0046] The text feature module uses the clip text editor to extract text features from the training set.

[0047] In the above technical solution, the clip image editor is used to replace image features from objects in the training set, which can extract high-level semantic information of the image. Combined with the semantic information of the text features, it can reflect the association characteristics of the image data and text data, thereby improving the naturalness and coherence of the object after replacement in the process of object replacement, and improving the coordination between the object and the video scene.

[0048] Furthermore, the object replacement module replaces the object in the video according to the extracted video frame features, object replacement image features and text features, including:

[0049] Input the noisy video frame features, the denoised historical video frame features, the object replacement image features and the text features into the object replacement module;

[0050] Based on the input noisy video frame features, denoised historical video frame features, object replacement image features and text features, the object replacement module uses the conditional diffusion method to iteratively denoise the video frame features and outputs the denoised video frame features, which can be expressed as:

[0051] Z t =Z′ t -∈ t ;

[0052] According to the object replacement image features and text features, the objects in the denoised video frame features are replaced, and the VAE decoder is used to decode the video frame features after object replacement to output the video after object replacement.

[0053] Among them, Z′ t represents the video frame feature with noise at time step t, Z t Represents the denoised video frame features.

[0054] In the above technical solution, based on the input noisy video frame features, the denoised historical video frame features, the object replacement image features and the text features, the object replacement module uses the conditional diffusion method to iteratively denoise the video frame features, which can make the potential representation in the data closer and closer to the original clean data, while ensuring that the generated content conforms to the text description and picture features. Finally, based on the object replacement image features and text features, the objects in the denoised video frame features are replaced, and the VAE decoder is used to decode the video frame features after object replacement, and output the video after object replacement, which effectively locates and tracks the target object, and maintains the naturalness and coherence after object replacement, thereby improving the coordination between the object and the video scene during the object replacement process.

[0055] Furthermore, the process of setting a loss function to optimize the parameters of the object replacement model in the video after each round of training includes:

[0056] Calculate the difference loss between video frames, the expression is:

[0057]

[0058] Calculate the optical flow consistency loss of consecutive frames in the video frame, the expression is:

[0059]

[0060] Calculate the video temporal coherence loss, the expression is:

[0061]

[0062] Based on the video frame difference loss, optical flow consistency loss, and video temporal coherence loss, the total loss of each round of optimization training of the object replacement model in the video is calculated. The expression is:

[0063] L total =ɑ·L recon +β·L flow +γ·L cosine ;

[0064] Optimize the parameters of the object replacement model in training based on the total loss of each round of optimization training of the object replacement model in the video;

[0065] Among them, χ i represents the real image, y i represents the image after VAE decoding, N represents the total number of frames, ||·||2 represents the L2 norm, T represents the total number of frames of the extracted video, I t Represents the t-th frame image, Flow(I t , I t+1 ) represents the real optical flow from the tth frame to the t+2th frame, Flow′(I t , I t+1 ) represents the optical flow of the video frames generated by the model from the t-th frame to the t+2-th frame, e1 represents the feature vector of the generated video after being encoded by ResNet+LSTM, e2 represents the feature vector of the real video after being encoded by ResNet+LSTM, e1·e2 represents the dot product of the feature vectors of the generated video and the real video, ||e1||||e2|| represents the modulus of the feature vectors of the generated video and the real video, ɑ, β, and γ represent the weights of their respective loss terms.

[0066] In the above technical solution, by setting different loss functions and optimizing the parameters of the object replacement model in the video during the object replacement training process, the target object can be effectively and accurately located and tracked, and the naturalness and continuity of the object after replacement can be maintained, thereby improving the coordination between the object and the video scene during the object replacement process.

[0067] A system for replacing objects in a video, the system comprising:

[0068] A data acquisition module, used for acquiring video image data;

[0069] A data processing module, configured to pre-process the video image data;

[0070] A feature extraction module is used to extract video frame features, object replacement image features and text features based on the preprocessed video image data;

[0071] The object replacement module is used to replace objects in the video based on the extracted video frame features, object replacement image features and text features.

[0072] Compared with the prior art, the present invention has the following beneficial effects:

[0073] The present invention proposes a method and system for replacing objects in videos. By preprocessing the acquired video image data, the practicality and reliability of the data can be improved. The extracted video frame features, object replacement image features and text features can intuitively reflect the positioning and tracking information of the target object from the data. Then, according to the correlation between different features, the object in the video is replaced, and the naturalness and coherence of the object after replacement are maintained, thereby improving the coordination between the object and the video scene during the object replacement process. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 A flowchart of the steps of a method for replacing an object in a video provided in an embodiment of the present application;

[0075] Figure 2 A schematic diagram of the structure of the object replacement model in the video provided in the embodiment of the present application;

[0076] Figure 3 A schematic structural diagram of a system for replacing objects in a video provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. Preferred embodiments of the present invention are shown in the accompanying drawings. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present disclosure.

[0078] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0079] Example 1:

[0080] A method for object replacement in video, see Figure 1 , the method comprises the following steps:

[0081] Step S1: Acquire video image data and pre-process the video image data;

[0082] Step S2: extracting video frame features, object replacement image features, and text features based on the preprocessed video image data;

[0083] Step S3: replacing objects in the video based on the extracted video frame features, image features, and text features.

[0084] In step S1, the process of preprocessing the video image data includes:

[0085] De-noising and framing the video data in the video image data to extract a number of de-noised video frames;

[0086] Denoising and normalization processing are performed on the object replacement image data in the video image data, and text description is added to the object replacement image data.

[0087] It can be understood that denoising video data can reduce the complexity of the data processing process and improve the efficiency of data processing. Frame processing of video data can better locate and track target objects to improve the accuracy of locating and tracking target objects. Denoising and normalizing image data can reduce the complexity of data and remove noise in the image while maintaining image details. Adding text descriptions to object replacement image data can improve the feature representation ability of image data, thereby improving the practicality and reliability of image data.

[0088] In this embodiment, preprocessing of the acquired video image data can improve the practicality and reliability of the data; the extracted video frame features, object replacement image features and text features can intuitively reflect the positioning and tracking information of the target object from the data, and then replace the object in the video according to the correlation between different features, and maintain the naturalness and coherence of the object after replacement, thereby improving the coordination between the object and the video scene during the object replacement process.

[0089] Example 2:

[0090] This embodiment further illustrates steps S2 and S3 based on the first embodiment, as follows:

[0091] In step S2, see Figure 2 ,The process of extracting video frame features, image features and text features includes:

[0092] Step S21: Divide the preprocessed video image data into a training set and a validation set, and construct a video object replacement model, wherein the video object replacement model includes a video frame feature module, an object replacement image feature module, a text feature module, and an object replacement module;

[0093] Step S22: inputting the data in the training set into the constructed video object replacement model, and setting several rounds of iterative optimization training to optimize the parameters of the video object replacement model;

[0094] Step S23: extracting video frame features in the training set using a video frame feature module;

[0095] Step S24: extracting object replacement image features in the training set using the object replacement image feature module;

[0096] Step S25: extracting text features from the training set using a text feature module;

[0097] Step S26: The object replacement module replaces the object in the video according to the extracted video frame features, object replacement image features, and text features, and outputs the object replacement result;

[0098] Step S27: Use the validation set to verify the object replacement results after each round of training, and set the loss function to optimize the parameters of the object replacement model in the video after each round of training. When the preset iterative optimization training round ends or the loss function converges, the trained object replacement model in the video is obtained, and the trained object replacement model in the video is used to replace the video image data to be replaced, and the object replacement result is output.

[0099] In step S23, see Figure 2,The process of extracting video frame features in the training set using the video frame feature module includes:

[0100] Use VAE encoder to extract video frame features from each video frame;

[0101] Use an image segmentation method (SegmentAnything Model 2) to segment each video frame, extract the foreground mask and background mask corresponding to each video frame, and extract the corresponding foreground attenuation mask and background attenuation mask based on the extracted foreground mask and background mask. The area with a value close to 1 represents the foreground (target object), and the area with a value close to 0 represents the background;

[0102] Extracting the features of the noisy video frame according to the video frame features, the foreground attenuation mask and the background attenuation mask;

[0103] Among them, the foreground mask represents the object to be replaced, and the background mask represents the background of the image in the video frame.

[0104] Specifically, the process of extracting video frame features from each video frame using the VAE encoder includes:

[0105] Each frame of the video is encoded separately (the video is composed of a series of consecutive frames, and each frame is an independent image), and the video frame image is converted into a latent space representation through the VAE encoder, which is recorded as the video frame feature, which is a vector of fixed dimension. This vector captures the semantic information of the video frame.

[0106] Specifically, the process of extracting the corresponding foreground attenuation mask and background attenuation mask includes:

[0107] A distance transform algorithm (Euclidean distance transform or city block distance transform) is used to calculate the distance from each pixel in the image to the nearest foreground pixel, and a corresponding distance image is generated based on the distance transform result, where the value of each pixel represents its distance to the nearest foreground pixel;

[0108] Set the attenuation factor and determine the maximum and minimum values ​​of the attenuation factor; in this example, the attenuation factor of the foreground (nearest foreground pixels) is 1, while the attenuation factor of the background (furthest foreground pixels) is close to 0.

[0109] According to the distance image, the maximum and minimum values ​​of the attenuation factor, a linear attenuation function is used to map the distance to the value of the attenuation factor. The expression is:

[0110]

[0111] The foreground attenuation mask and background attenuation mask are extracted based on the attenuation factor value. The attenuation factor of the foreground pixel is 1, the attenuation factor of the background pixel is close to 0, and the pixels in between have an attenuation factor that decreases from 1 to 0 according to their distance to the foreground.

[0112] Among them, d represents the distance transformation result, D max Indicates the maximum distance value in the distance transformation.

[0113] Specifically, the process of extracting features of noisy video frames includes:

[0114] Set the noise level sequence σ1, σ2, σ3, ..., σ T ;

[0115] Based on the noise level sequence, the foreground attenuation mask and the background attenuation mask are noised using element-by-element multiplication (each noise added to the video frame feature will be fused with the attenuated foreground and background masks, so that the noise intensity added to the foreground area is the largest, while the noise in the background area decreases as the distance from the foreground increases). The expression is:

[0116] ∈ t =σ t ⊙M;

[0117] The noise is fused with the foreground attenuation mask and the background attenuation mask after the noise addition into the video frame features. For each step (from t to T), the current potential representation Z t-1 Initially, noise is added to the current potential representation to obtain the noisy video frame feature, which is expressed as:

[0118] Z′ t =Z t-1 +∈′ t ;

[0119] Where T is the total number of steps in the noise addition process, σ T =0 means no noise, σ1 is the largest noise category, ∈′ t Represents the noise used in the Gaussian distribution, ⊙ represents element-wise multiplication, M represents the foreground attenuation mask and the background attenuation mask, and t represents the time step.

[0120] It can be understood that by using the VAE encoder to extract video frame features from each video frame, the semantic information of the video frame can be effectively captured to convert the frame into a low-dimensional table of high-dimensional data; the foreground mask and background mask corresponding to each video frame are extracted, and based on the extracted foreground mask and background mask, the corresponding foreground attenuation mask and background attenuation mask are extracted, which can gradually introduce noise to each noise-free original video frame feature, simulating the gradual transition from data distribution to Gaussian noise distribution, while introducing randomness and diversity in the generation process. The method of gradually adding noise enables the model to capture the complex structure of the data during the generation process while maintaining control over the generation process. It can also maintain the integrity of the background elements in the process of replacing the target object, while maintaining the naturalness and coherence of the object after replacement, thereby improving the coordination between the object and the video scene during the object replacement process.

[0121] In step S24 and step S25, see Figure 2 ,The object replacement image feature module uses the clip image editor to replace image features from objects in the training set;

[0122] The text feature module uses the clip text editor to extract text features from the training set.

[0123] It can be understood that using the clip image editor to replace image features from objects in the training set can extract high-level semantic information of the image. Combined with the semantic information of the text features, it can reflect the association characteristics of the image data and text data, thereby improving the naturalness and coherence of the object after replacement, and improving the coordination between the object and the video scene.

[0124] In step S26, see Figure 2 The object replacement module replaces objects in the video according to the extracted video frame features, object replacement image features and text features. The process includes:

[0125] Input the noisy video frame features, the denoised historical video frame features, the object replacement image features and the text features into the object replacement module;

[0126] Based on the input noisy video frame features, denoised historical video frame features, object replacement image features and text features, the object replacement module uses the conditional diffusion method to iteratively denoise the video frame features and outputs the denoised video frame features, which can be expressed as:

[0127] Z t =Z′ t -∈ t ;

[0128] According to the object replacement image features and text features, the objects in the denoised video frame features are replaced, and the VAE decoder is used to decode the video frame features after object replacement to output the video after object replacement.

[0129] Among them, Z′ t represents the video frame feature with noise at time step t, Z t Represents the denoised video frame features.

[0130] Specifically, this method gradually restores the noisy video frame features to the noise-free video frame features (i.e., the process of generating video frames after the subject is replaced). Conditional Diffusion Models (CDMs) are used here. The core is a U-Net structure, a common neural network architecture for image and video generation. U-Net effectively combines contextual information and position information through its encoder-decoder structure.

[0131] During the denoising process, the features of all previous video frames, text features, and feature information of the image to be replaced are used as auxiliary information to assist in denoising the video frame. The detailed process is as follows:

[0132] For each step t, the denoised historical video frame features, text features, and image features are combined with the current video frame features Z′ t As the input of the denoising model, the noise ∈ t , this process can be described as:

[0133] ∈ t =f θ (Z′ t ,t,E text ,E img ,Z history )

[0134] Among them, Z′ t is the noisy video frame feature at time step t, f θ is a denoising model with parameter θ, which is used to predict the noise based on the noisy input and time step information, ∈ t is the predicted noise, E text ,E img Represent text and image features respectively, Z historyThe historical video frame output after denoising by the denoising model can be understood as the sum of all video frame features before the current video frame timestamp, calculating the L2 norm of the summed vector (i.e., the square root of the sum of the squares of the vector elements), and then dividing the elements of each dimension in the summed vector by this norm to obtain the normalized historical video frame features. (The purpose of using this feature is to enable the model to utilize the video frame features of previous time steps.)

[0135] The model then uses this predicted noise to update the video frame features to gradually restore clean data, Z t is the denoised video frame feature:

[0136] Z t =Z′ t -∈ t

[0137] The denoising process is iterated until all T steps have been completed. Each step gradually reduces the noise, bringing the latent representation closer and closer to the original clean data while ensuring that the generated content conforms to the text description and image features. After decoding is complete, the video frame features are decoded back into the video frame image through the VAE decoder.

[0138] It can be understood that according to the input noisy video frame features, the denoised historical video frame features, the object replacement image features and the text features, the object replacement module uses the conditional diffusion method to iteratively denoise the video frame features, which can make the potential representation in the data closer and closer to the original clean data, while ensuring that the generated content conforms to the text description and picture features. Finally, according to the object replacement image features and text features, the objects in the denoised video frame features are replaced, and the VAE decoder is used to decode the video frame features after object replacement, and output the video after object replacement, which effectively locates and tracks the target object, and maintains the naturalness and coherence after object replacement, thereby improving the coordination between the object and the video scene during the object replacement process.

[0139] In step S27, the process of setting a loss function to optimize the parameters of the object replacement model in the video after each round of training includes:

[0140] Calculate the difference loss between video frames, prepare the video frame images decoded by VAE and the real video frame images at the corresponding time points, and ensure that they have the same size and format.

[0141] Calculate the reconstruction loss, which measures the pixel-level difference between each video frame generated by the model and the real video frame. The expression is:

[0142]

[0143] Calculate the optical flow consistency loss of consecutive frames in the video frame. This loss function uses optical flow to measure the motion consistency between consecutive frames. The expression is:

[0144]

[0145] Calculate the video temporal coherence loss, the expression is:

[0146]

[0147] Specifically, a pre-trained ResNet+LSTM video encoder is used, and the model parameters remain unchanged. All video frames of the generated and real videos are input into the model in chronological order. The ResNet+LSTM is used to encode the temporal and semantic information between the video frames, converting them into fixed-dimensional image vectors. These image vectors contain not only the static information of the frame but also the temporal information between the frames. These image vectors are then input into the LSTM model to obtain feature vectors that represent the entire video content. Finally, the cosine similarity of the two video feature vectors is calculated to measure the similarity between the generated and real videos while maintaining temporal consistency and coherence between the two videos.

[0148] Based on the video frame difference loss, optical flow consistency loss, and video temporal coherence loss, the total loss of each round of optimization training of the object replacement model in the video is calculated. The expression is:

[0149] L total =α·L recon +β·L flow +γ·L cosine ;

[0150] Optimize the parameters of the object replacement model in training based on the total loss of each round of optimization training of the object replacement model in the video;

[0151] Among them, x i represents the real image, y i represents the image after VAE decoding, N represents the total number of frames, ||·||2 represents the L2 norm, T represents the total number of frames of the extracted video, I t Represents the t-th frame image, Flow(I t ,I t+1 ) represents the real optical flow from the tth frame to the t+2th frame, Flow′(I t ,I t+1) represents the optical flow of the model-generated video frames from frame t to frame t+2, e1 represents the feature vector of the generated video after ResNet+LSTM encoding, e2 represents the feature vector of the real video after ResNet+LSTM encoding, e1·e2 represents the dot product of the feature vectors of the generated and real videos, ||e1||||e2|| represents the modulus of the feature vectors of the generated and real videos, α, β, and γ represent the weights of the respective loss terms, which can be adjusted according to actual conditions. Here, they are set to 2:3:5, with the video temporal coherence loss having the highest weight. By optimizing the model parameters and minimizing the loss function through backpropagation and gradient descent algorithms, the generated video is ensured to be as close as possible to the real video in multiple aspects, thereby improving the quality of video generation. The trained object replacement model can be applied to product display videos based on text prompts, replacing products in the videos and adjusting the texture and style of objects in the videos based on the prompt description.

[0152] It can be understood that by setting different loss functions and optimizing the parameters of the object replacement model in the video during the object replacement training process, the target object can be accurately located and tracked, and the naturalness and continuity of the object after replacement can be maintained, thereby improving the coordination between the object and the video scene during the object replacement process.

[0153] In this embodiment, after training and verification, the constructed object replacement model in the video can efficiently identify and analyze the input video image data, thereby effectively and accurately positioning and tracking the target object, and maintaining the naturalness and continuity of the object after replacement, thereby improving the coordination between the object and the video scene during the object replacement process.

[0154] Example 3:

[0155] A system for object replacement in videos, see Figure 3 , the system comprising:

[0156] A data acquisition module, used for acquiring video image data;

[0157] A data processing module, configured to pre-process the video image data;

[0158] A feature extraction module is used to extract video frame features, object replacement image features and text features based on the preprocessed video image data;

[0159] The object replacement module is used to replace objects in the video based on the extracted video frame features, object replacement image features and text features.

[0160] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention's description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for replacing objects in a video, characterized in that: The method comprises the following steps: Acquiring video image data and preprocessing the video image data; Extract video frame features, object replacement image features, and text features based on the preprocessed video image data; Replace objects in the video based on the extracted video frame features, object replacement image features, and text features; The process of extracting video frame features, image features, and text features includes: The preprocessed video image data is divided into a training set and a validation set, and a video object replacement model is constructed, wherein the video object replacement model includes a video frame feature module, an object replacement image feature module, a text feature module and an object replacement module; Input the data in the training set into the constructed object replacement model in the video, and set up several rounds of iterative optimization training to optimize the parameters of the object replacement model in the video; Use the video frame feature module to extract video frame features in the training set; Use the object replacement image feature module to extract the object replacement image features in the training set; Use the text feature module to extract text features in the training set; The object replacement module replaces objects in the video based on the extracted video frame features, object replacement image features, and text features, and outputs the object replacement results; The object replacement results after each round of training are verified using the validation set, and a loss function is set to optimize the parameters of the object replacement model in the video after each round of training. When the preset iterative optimization training round ends or the loss function converges, the trained object replacement model in the video is obtained. The trained object replacement model is used to replace the video image data to be replaced, and the object replacement result is output; The process of extracting video frame features from the training set using the video frame feature module includes: Use VAE encoder to extract video frame features from each video frame; Performing image segmentation on each video frame using an image segmentation method, extracting a foreground mask and a background mask corresponding to each video frame, and extracting a corresponding foreground attenuation mask and a background attenuation mask based on the extracted foreground mask and background mask; Extracting the features of the noisy video frame according to the video frame features, the foreground attenuation mask and the background attenuation mask; Among them, the foreground mask represents the object to be replaced, and the background mask represents the background of the image in the video frame; The object replacement module replaces objects in the video based on the extracted video frame features, object replacement image features, and text features. The process includes: Input the noisy video frame features, the denoised historical video frame features, the object replacement image features and the text features into the object replacement module; Based on the input noisy video frame features, denoised historical video frame features, object replacement image features and text features, the object replacement module uses the conditional diffusion method to iteratively denoise the video frame features and outputs the denoised video frame features, which can be expressed as: WITH t =Z' t -∈ t ; According to the object replacement image features and text features, the object in the denoised video frame features is replaced, and the VAE decoder is used to decode the video frame features after object replacement, and output the video after object replacement; Among them, Z' t represents the video frame feature with noise at time step t, Z t Represents the denoised video frame features, ∈ t represents the noise of the prediction.

2. The method for replacing objects in a video according to claim 1, wherein: The process of preprocessing the video image data includes: De-noising and framing the video data in the video image data to extract a number of de-noised video frames; Denoising and normalization processing are performed on the object replacement image data in the video image data, and text description is added to the object replacement image data.

3. The method for replacing objects in a video according to claim 1, wherein: The process of extracting the corresponding foreground attenuation mask and background attenuation mask includes: The distance transform algorithm is used to calculate the distance between each pixel in the image and the nearest foreground pixel, and the corresponding distance image is generated based on the distance transform result; Set the attenuation factor and determine the maximum and minimum values ​​of the attenuation factor; According to the distance image, the maximum and minimum values ​​of the attenuation factor, a linear attenuation function is used to map the distance to the value of the attenuation factor. The expression is: The foreground attenuation mask and the background attenuation mask are extracted according to the value of the attenuation factor; Among them, d represents the distance transformation result, D max Indicates the maximum distance value in the distance transformation.

4. The method for replacing objects in a video according to claim 1, wherein: The process of extracting features from noisy video frames includes: Set the noise level sequence σ1,σ2,σ3,....,σ T ; Based on the noise level sequence, the foreground attenuation mask and the background attenuation mask are noised using element-by-element multiplication. The expression is: ∈' t =s t ⊙M; The noise and the foreground attenuation mask and background attenuation mask after noise processing are fused into the video frame features to obtain the noisy video frame features, which are expressed as: WITH' t =Z t-1 +∈' t ; Where T is the total number of steps in the noise addition process, σ T =0 means no noise, σ1 is the largest noise category, ∈' t Represents the noise used in the Gaussian distribution, ⊙ represents element-wise multiplication, M represents the foreground attenuation mask and the background attenuation mask, and t represents the time step.

5. The method for replacing objects in a video according to claim 1, wherein: The object replacement image feature module uses the clip image editor to extract the object replacement image features in the training set; The text feature module uses the clip text editor to extract text features in the training set.

6. The method for replacing objects in a video according to claim 1, wherein: The process of setting the loss function to optimize the parameters of the object replacement model in each round of training includes: Calculate the difference loss between video frames, the expression is: Calculate the optical flow consistency loss of consecutive frames in the video frame, the expression is: Calculate the video temporal coherence loss, the expression is: Based on the video frame difference loss, optical flow consistency loss, and video temporal coherence loss, the total loss of each round of optimization training of the object replacement model in the video is calculated. The expression is: L total =α·L recon +β·L flow +γ·L cosine ; Optimize the parameters of the object replacement model in training based on the total loss of each round of optimization training of the object replacement model in the video; Among them, x i represents the real image, y i represents the image after VAE decoding, N represents the total number of frames, ||·||2 represents the L2 norm, T represents the total number of frames of the extracted video, I t Represents the t-th frame image, Flow(I t ,I t+1 ) represents the real optical flow from the tth frame to the t+2th frame, Flow'(I t ,I t+1 ) represents the optical flow of the video frames generated by the model from the t-th frame to the t+2-th frame, e1 represents the feature vector of the generated video after being encoded by ResNet+LSTM, e2 represents the feature vector of the real video after being encoded by ResNet+LSTM, e1·e2 represents the dot product of the feature vectors of the generated video and the real video, ||e1||||e2|| represents the modulus of the feature vectors of the generated video and the real video, and α, β, and γ represent the weights of their respective loss terms.

7. A system for replacing objects in a video, the system being based on the method according to any one of claims 1 to 6, characterized in that: The system comprises: A data acquisition module, used for acquiring video image data; A data processing module, configured to pre-process the video image data; A feature extraction module is used to extract video frame features, object replacement image features and text features based on the preprocessed video image data; The object replacement module is used to replace objects in the video based on the extracted video frame features, object replacement image features and text features.

Images