Video processing method and apparatus, readable storage medium and program product

Abstract

The application relates to the technical field of computer vision, and provides a video processing method and device, a readable storage medium and a program product. The method comprises the following steps: obtaining a reference image and a mask video according to a first video; obtaining a video text feature according to semantic description information of video content of the first video; performing context coding processing on a reference image feature of the reference image, the video text feature, a mask video feature of the mask video and a noise video of the first video to obtain a context coding feature; in a process of performing layer-by-layer denoising processing based on the reference image feature, the video text feature and the noise video, each layer of denoising processing result is fused with the context coding feature; and a second video after adjustment of a target part is obtained according to a denoising processing result of a last layer. The method can guarantee the consistency of a non-redrawing area and the first video and the consistency of the overall style of the second video.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a video processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of deep learning technology, diffusion models have matured in the field of computer vision and are widely used in video processing scenarios, such as various visual tasks, including complex scenarios such as video content generation and editing.

[0003] Traditional video processing methods often rely on large-scale diffusion models, exhibiting strong controllability and generalization capabilities. For example, text-guided video processing methods have been applied to tasks such as video content enhancement, restoration, and style transfer.

[0004] However, traditional video processing methods may result in visual disharmony or unsatisfactory shapes of target parts, making it difficult to ensure consistency between non-redrawn areas and the original video, and also making it difficult to ensure the consistency of the overall style of the generated video. Summary of the Invention

[0005] Based on this, this application provides a video processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product, which can ensure the consistency between the non-redrawn area and the first video, as well as the consistency of the overall style of the second video.

[0006] On the one hand, this application provides a video processing method, including:

[0007] A reference image is determined from a plurality of frames included in the first video; the first video includes a plurality of frames, each frame including the target region;

[0008] Based on the semantic description information of the video content of the first video, video text features are obtained;

[0009] Based on the masking process applied to the target area, the masked video of the first video is obtained;

[0010] Context encoding is performed on the reference image features of the reference image, the video text features, the mask video features of the mask video, and the noisy video of the first video to obtain context encoded features;

[0011] In the process of performing layer-by-layer denoising based on reference image features, video text features, and noisy video, the denoising result of each layer is fused with the context coding features;

[0012] Based on the denoising results of the last layer, a second video is obtained after adjusting the target area.

[0013] In one embodiment, obtaining video text features based on semantic description information of the video content of the first video includes:

[0014] Input the first video into the multimodal large model to obtain semantic description information of the video content of the first video;

[0015] The semantic description information is extracted using a text encoder based on a diffusion model to obtain video text features.

[0016] In one embodiment, the step of performing layer-by-layer denoising based on reference image features, the video text features, and noisy video, involves fusing the denoising result of each layer with the context coding features, including:

[0017] The denoising backbone network of the reference image features, the video text features, and the noise video input diffusion model;

[0018] The denoising backbone network is configured to: process the reference image features, the video text features, and the noisy video through its first-layer denoising unit to obtain a first-layer denoising result; fuse the first-layer denoising result with the context coding features to obtain a first-layer denoising fusion result; process the previous-layer denoising fusion result through the non-first-layer denoising unit of the denoising backbone network to obtain a non-first-layer denoising result; and fuse the non-first-layer denoising result with the context coding features to obtain a non-first-layer denoising fusion result.

[0019] In one embodiment, the denoising process on the reference image features, the video text features, and the noisy video to obtain the first-layer denoising result includes:

[0020] Noisy video is subjected to feature processing to obtain noisy features;

[0021] Based on the reference image features and the video text features, the noisy features are conditionally calibrated to obtain the noisy calibrated features;

[0022] Based on the noisy calibration features, the first-layer denoising result is obtained.

[0023] In one embodiment, obtaining the first-layer denoising result based on the noisy calibration features includes:

[0024] The noisy calibration features are input into the low-rank adaptation unit of the diffusion model to obtain the low-rank increment.

[0025] Based on the low-rank increment, the noisy calibration feature is corrected to obtain the corrected noisy calibration feature.

[0026] Based on the corrected noisy calibration features, the first-layer denoising result is obtained.

[0027] In one embodiment, determining the reference image among the plurality of frames included in the first video includes:

[0028] Select the target frame image from the several frames included in the first video;

[0029] The target frame image is input to an image generation model, which enhances the target region in the target frame image to obtain a reference image.

[0030] In one embodiment, obtaining the noisy video of the first video includes:

[0031] Obtain a set noise intensity; the set noise intensity is lower than the pure noise intensity.

[0032] Noise is gradually added to the first video until a set noise level is reached, resulting in a noisy video of the first video.

[0033] On the one hand, this application also provides a video processing apparatus, including:

[0034] A reference image acquisition module is used to determine a reference image from several frames of images included in the first video; the first video includes several frames of images, and each frame of images includes a target part.

[0035] The video text feature acquisition module is used to obtain video text features based on the semantic description information of the video content of the first video;

[0036] The mask video acquisition module is used to obtain the mask video of the first video based on the mask processing for the target area;

[0037] The context coding feature acquisition module is used to perform context coding processing on the reference image features of the reference image, the video text features, the mask video features of the mask video, and the noisy video of the first video to obtain context coding features;

[0038] The fusion module is used to fuse the denoising result of each layer with the context coding features during the layer-by-layer denoising process based on reference image features, the video text features, and noisy video.

[0039] The second video acquisition module is used to obtain a second video after adjusting the target part based on the denoising processing result of the last layer.

[0040] In another aspect, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the methods described above.

[0041] In one aspect, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, the computer program being executed by a processor using the methods described above.

[0042] In one aspect, this application also provides a computer program product. The computer program product includes a computer program that is executed by a processor using the methods described above.

[0043] The aforementioned video processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product determine a reference image from several frames of images included in a first video; the first video includes several frames of images, each frame including a target region; video text features are obtained based on semantic description information of the video content of the first video; a masked video of the first video is obtained based on masking processing of the target region; context encoding processing is performed on the reference image features of the reference image, the video text features, the masked video features of the masked video, and the noisy video of the first video to obtain context encoded features; during the layer-by-layer denoising process based on the reference image features, video text features, and noisy video, the denoising result of each layer is fused with the context encoded features; and a second video after adjusting the target region is obtained based on the denoising result of the last layer. This application performs context encoding on reference image features, video text features, mask video features, and noisy video of the first video to obtain context-encoded features. The mask video features are integrated into the context-encoded features. During the layer-by-layer denoising process based on reference image features, video text features, and noisy video, the denoising results of each layer are fused with the context-encoded features. This enables separate modeling of the background context and the generated target region. The context-encoded features can be added to the feature representation of non-critical regions, maintaining consistency between non-masked regions and the first video. This avoids interference with the originally stable and important background or subject structure in the first video, thus better preserving the overall scene layout and visual style of the first video and ensuring consistency between non-redrawn regions and the first video, thereby improving the structural stability and semantic consistency of the second video. Furthermore, during video processing, using reference image features as a reference for adjusting the intensity and style of the target region provides higher accuracy and visual consistency, ensuring the overall stylistic consistency of the second video. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is an application environment diagram of a video processing method in one embodiment;

[0046] Figure 2 This is a flowchart illustrating a video processing method in one embodiment;

[0047] Figure 3 This is a schematic diagram of the structure of a low-rank adaptation unit in one embodiment;

[0048] Figure 4 This is a schematic diagram of the overall structure of a video processing method in one embodiment;

[0049] Figure 5 This is a structural block diagram of a video processing device in one embodiment;

[0050] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and beneficial effects of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] The video processing method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 obtains a reference image, video text features, and a masked video based on the first video; it performs context coding on the reference image features of the reference image, the video text features, the masked video features of the masked video, and the noisy video of the first video to obtain context-coded features; during the layer-by-layer denoising process based on the reference image features, video text features, and noisy video, the denoising result of each layer is fused with the context-coded features; based on the denoising result of the final layer, a second video after adjusting the target area is obtained.

[0053] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0054] In one exemplary embodiment, such as Figure 2 As shown, a video processing method is provided, which is applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps S201 to S206. Wherein:

[0055] Step S201: Determine a reference image from the several frames of images included in the first video; the first video includes several frames of images, and each frame of images includes the target part.

[0056] The video of the target area to be adjusted can be called the first video.

[0057] The first video consists of several frames, each of which includes a target body part. The target body part can be a specific area of ​​a target object within each frame. For example, the target body part could be the chest area of ​​a person in each frame.

[0058] One frame can be randomly selected from several frames in the first video for target area enhancement to obtain a reference image.

[0059] Step S202: Obtain video text features based on the semantic description information of the video content of the first video.

[0060] Based on the first video, semantic description information of the video content of the first video can be obtained.

[0061] Feature extraction can be performed on the above semantic description information to obtain video text features.

[0062] Step S203: Based on the masking process applied to the target area, obtain the masked video of the first video.

[0063] The first video is input into the segmentation model so that the segmentation model can segment each frame of the first video. The target region of each frame of the first video is extracted by the segmentation algorithm, and the corresponding target region mask is generated. The target region mask is then applied to the first video, and the non-masked regions are retained to obtain the masked video of the first video.

[0064] The target area mask obtained from segmentation may be inaccurate. In order to better cover the target area that needs to be adjusted, the target area mask can be etched to expand the mask area.

[0065] Step S204: Perform context coding on the reference image features of the reference image, the video text features, the mask video features of the mask video, and the noisy video of the first video to obtain context-coded features.

[0066] Feature extraction can be performed on the reference image to obtain the reference image features.

[0067] Feature extraction can be performed on the masked video to obtain the masked video features.

[0068] Based on the implicit guidance mechanism of the video content of the first video, the structure and style information of the first video can be encoded into noise to obtain the noisy video of the first video.

[0069] The reference image features, video text features, masked video features, and noisy video of the first video can be input into the context encoder of the diffusion model, so that the context encoder performs context encoding processing on the reference image features, video text features, masked video features, and noisy video of the first video, and fuses them to generate context-coded features with semantic consistency.

[0070] Step S205: In the process of performing layer-by-layer denoising based on reference image features, video text features and noisy video, the denoising results of each layer are fused with the context coding features.

[0071] Reference image features, video text features, and noisy video can be input into the denoising backbone network of the diffusion model. This backbone network then performs layer-by-layer denoising on these features using multiple denoising units, culminating in the final denoising result. The denoising result of each layer is fused with context-coded features and used as input to the next denoising unit.

[0072] Step S206: Based on the denoising results of the last layer, a second video after adjusting the target area is obtained.

[0073] The denoising result of the last layer can be input into the decoder of the diffusion model so that the decoder can perform a decoding operation on the denoising result of the last layer to obtain a second video after adjusting the target parts of each frame of the first video.

[0074] In the aforementioned video processing method, context encoding is performed on the reference image features of the reference image, video text features, mask video features of the mask video, and noisy video of the first video to obtain context-encoded features. These mask video features are then integrated into the context-encoded features. During the layer-by-layer denoising process based on the reference image features, video text features, and noisy video, the denoising results of each layer are fused with the context-encoded features. This allows for separate modeling of the background context and the generated target region. The context-encoded features can be added to the feature representation of non-critical regions, maintaining consistency between non-masked regions and the first video. This avoids interference with the originally stable and important background or main structure in the first video, thus better preserving the overall scene layout and visual style of the first video and ensuring consistency between non-redrawn regions and the first video, thereby improving the structural stability and semantic consistency of the second video. Furthermore, using the reference image features as a reference for adjusting the intensity and style of the target region during video processing provides higher accuracy and visual consistency, ensuring the overall stylistic consistency of the second video.

[0075] In one embodiment, video text features are obtained based on the semantic description information of the video content of the first video. The specific steps are as follows: input the first video into a multimodal large model to obtain the semantic description information of the video content of the first video; extract features from the semantic description information through the text encoder of the diffusion model to obtain the video text features.

[0076] The first video can be input into a multimodal large model so that the multimodal large model can extract features from the first video and obtain semantic description information of the video content of the first video.

[0077] Semantic description information can be input into the text encoder of the diffusion model, so that the text encoder can extract features from the semantic description information to obtain video text features.

[0078] In this embodiment, the first video is input into a multimodal large model to obtain semantic description information of the video content of the first video; the text encoder of the diffusion model is used to extract features from the semantic description information to obtain video text features, so as to control the overall semantic and stylistic consistency of the second video during the second video generation process.

[0079] In one embodiment, during the layer-by-layer denoising process based on reference image features, video text features, and noisy video, the denoising result of each layer is fused with the context coding features. The specific steps are as follows: The reference image features, video text features, and noisy video are input into the denoising backbone network of the diffusion model; wherein, the denoising backbone network is used to denoise the reference image features, video text features, and noisy video through the first-layer denoising unit of the denoising backbone network to obtain the first-layer denoising result; the first-layer denoising result is fused with the context coding features to obtain the first-layer denoising fusion result; the previous-layer denoising fusion result is denoised through the non-first-layer denoising unit of the denoising backbone network to obtain the non-first-layer denoising result; the non-first-layer denoising result is fused with the context coding features to obtain the non-first-layer denoising fusion result.

[0080] This can be used to construct a denoising backbone network for a DiffusionTransformer (DiT) model that incorporates reference image features, video text features, and noisy video input. The denoising backbone network consists of six denoising units, each of which is a Transformer Encoder layer adapted to the requirements of the diffusion model.

[0081] The noise distribution of this step can be predicted by the first-layer denoising unit of the denoising backbone network based on the features of the reference image, the features of the video text, and the noisy video. Then, the noise distribution of the current noisy video can be iteratively adjusted according to the set sampling strategy to reduce the noise ratio of the current noisy video and obtain the first-layer denoising result. The first-layer denoising result can be fused with the context coding features to obtain the first-layer denoising fusion result.

[0082] The input to the first-layer denoising unit can include three parts. The first part can include noisy video. The second part can include reference image features and video text features. The third part can include a time series encoding T.

[0083] The formula for calculating the loss function of the first-layer denoising unit is shown in equation (1).

[0084] (1)

[0085] in, This represents Gaussian noise obtained through random sampling during the noise addition process. This represents the first part of the input, namely the noisy video. This represents the input for the second part, namely, the reference image features and the video text features. This represents the third part of the input, namely the encoding T of a time series. This represents a noise prediction model. This represents the mean squared error loss function.

[0086] The noise distribution of this step can be predicted by the non-first-layer denoising unit of the denoising backbone network based on the denoising fusion result of the previous layer. Then, the current noisy video can be iteratively adjusted according to the set sampling strategy to reduce the noise ratio of the current noisy video and obtain the non-first-layer denoising processing result. The non-first-layer denoising processing result is fused with the context coding features to obtain the non-first-layer denoising fusion result.

[0087] In this embodiment, the denoising results of each layer are fused with context coding features, which can achieve separate modeling of background context and target area generation. Context coding features can be added to the feature representation of non-critical areas, maintaining consistency between non-masked areas and the first video. This avoids interference with the originally stable and important background or main structure in the first video, thereby better preserving the overall scene layout and visual style of the first video and ensuring consistency between non-redrawn areas and the first video, thus improving the structural stability and semantic consistency of the second video.

[0088] In one embodiment, the reference image features, video text features, and noisy video are denoised to obtain the first-layer denoising result. The specific steps are as follows: the noisy video is processed to obtain noisy features; based on the reference image features and video text features, the noisy features are conditionally calibrated to obtain noisy calibration features; and the first-layer denoising result is obtained based on the noisy calibration features.

[0089] Feature extractors can be used to extract key information (such as spatiotemporal structure and semantic features) from noisy videos to obtain noisy features.

[0090] Based on the visual standards of the reference image features and the semantic standards of the video text features, invalid noise that is unrelated to the reference image features and video text features can be filtered out from the noisy features. At the same time, the noisy features can be made to fit the structure and semantics of the first video. For example, it can be ensured that the denoised video still retains the object shape in the reference image and matches the scene described in the text, thus obtaining the noisy calibration features.

[0091] Based on the noise-containing calibration characteristics, the noise distribution of this step can be predicted. Then, based on the set sampling strategy and the noise distribution of this step, the current noisy video can be iteratively adjusted to reduce the noise ratio of the current noisy video and obtain the first-layer denoising result.

[0092] In this embodiment, feature processing is performed on the noisy video to obtain noisy features, which can convert pixel-level noise in the noisy video into feature-level noise, facilitating subsequent targeted processing. Based on reference image features and video text features, the noisy features are conditionally calibrated to obtain noisy calibration features, which can make the noisy calibration features fit the structure and semantics of the first video, avoiding deviation of subsequent denoising from the target. Based on the noisy calibration features, the first-layer denoising processing result is obtained, which can remove the remaining noise in the noisy calibration features.

[0093] In one embodiment, the first-layer denoising result is obtained based on the noisy calibration features. The specific steps are as follows: input the noisy calibration features into the low-rank adaptation unit of the diffusion model to obtain the low-rank increment; correct the noisy calibration features based on the low-rank increment to obtain the corrected noisy calibration features; and obtain the first-layer denoising result based on the corrected noisy calibration features.

[0094] Low-rank adaptive unit structure such as Figure 3 As shown in the diagram. Here, h represents the input to the low-rank adaptation unit, W represents the pre-trained weight parameters, and A and B represent the trainable model parameters used for fine-tuning. denoted by scaling factor. h1 represents the output result of the original pre-trained weights after fine-tuning, which can be expressed as equation (2).

[0095] (2)

[0096] This embodiment further designs the low-rank adaptation unit so that the generated features are applied to the target region and not to non-target regions. The loss function formula during training is shown in equation (3).

[0097] (3)

[0098] in, This represents the loss function calculated when training low-rank adaptive units. express The original features of the masked region at any given time, i.e., the original features of the target region. This represents Gaussian noise obtained through random sampling during the noise addition process. Indicates noisy calibration features. The encoding T represents a time series. This represents the noise prediction model. The loss function ensures that the target region in the generated second video maintains consistency with the target region features in the reference image.

[0099] Noisy calibration features can be input into the low-rank adaptation (LoRA) unit of the diffusion model, so that the low-rank adaptation unit can obtain a lightweight correction signal, i.e., low-rank increment, by learning the features common to the target part.

[0100] The noisy calibration features can be corrected based on the low-rank increment, resulting in the corrected noisy calibration features. Based on the corrected noisy calibration features, the noise distribution of this step can be predicted. Then, based on the set sampling strategy and the noise distribution of this step, the current noisy video is iteratively adjusted to reduce the noise ratio of the current noisy video, thus obtaining the first-layer denoising result.

[0101] Furthermore, during the process of obtaining the non-first-layer denoising results, the previous-layer denoising fusion results can be input into the low-rank adaptation unit of the diffusion model, so that the low-rank adaptation unit can obtain a lightweight correction signal, i.e., low-rank increment, by learning the common features of the target part.

[0102] The previous layer's denoising and fusion result can be corrected based on the low-rank increment, resulting in a corrected previous layer denoising and fusion result. Based on this corrected result, the noise distribution for this step can be predicted. Then, based on the set sampling strategy and the noise distribution for this step, the current noisy video can be iteratively adjusted to reduce the noise ratio, thus obtaining the non-first layer denoising processing result.

[0103] In this embodiment, a low-rank adaptive unit is used to perform lightweight correction on the noisy calibration features to control the generation results in the target area, so that the target area is consistent with the target area features in the reference image in terms of shape and style. Under the premise of keeping the diffusion model parameters unchanged, the low-rank adaptive unit can be used to fine-tune the diffusion model, enhance the diffusion model's ability to capture and reproduce the target area features in the reference image, and ensure that the generated target area is highly consistent with the reference image in detail.

[0104] In one embodiment, a reference image is determined from a plurality of frames included in the first video. The specific steps are as follows: a target frame image is selected from a plurality of frames included in the first video; the target frame image is input into an image generation large model so that the image generation large model enhances the target part in the target frame image to obtain a reference image.

[0105] One frame can be randomly selected from several frames included in the first video as the target frame. For example, the first frame of the first video can be used as the target frame.

[0106] The target frame image can be input into the image generation large model, which can then enhance the target region in the target frame image to obtain a reference image.

[0107] This reference image can serve as a reference for the generation intensity and style of the target part in the diffusion model, providing higher accuracy and visual consistency, thereby ensuring the overall style consistency of the second video.

[0108] In one embodiment, the noise video of the first video is obtained by the following steps: obtaining a set noise intensity; setting the noise intensity to be lower than the pure noise intensity; gradually adding noise to the first video up to the set noise intensity to obtain the noise video of the first video.

[0109] When a traditional diffusion model performs forward diffusion on the first video, it adds noise to the first video to make it a pure noise video. During this process, the noise intensity of the video in the intermediate stage of adding noise is lower than the noise intensity of the pure noise video.

[0110] In order to better maintain the consistency of the target part generation result with the first video in the overall style during the subsequent layer-by-layer denoising process, this embodiment does not add noise to the first video to a pure noise video. Instead, it adds noise to the first video step by step to a set noise intensity σ based on the denoising diffusion probabilistic model. This set noise intensity σ is lower than the pure noise intensity. It is equivalent to using the video in the middle stage of adding noise as the noise video of the first video, so that the noise video of the first video can retain some low-frequency structural information of the first video.

[0111] Among them, the noise addition mechanism based on the denoising diffusion probability model, the process of gradually adding noise to the first video is shown in Equation (4).

[0112] (4)

[0113] In the formula, This represents the video with added noise at time step t. This represents the video before noise addition at time step t. This represents the cumulative noise attenuation coefficient corresponding to time step t. Let represent the Gaussian noise obtained by random sampling, as shown in equation (5).

[0114] (5)

[0115] In the formula, Indicates a Gaussian distribution. Let represent the covariance matrix.

[0116] In this embodiment, noise is gradually added to the first video up to a set noise intensity, resulting in a noisy video of the first video. This set noise intensity is lower than the pure noise intensity. The noisy video of the first video can retain some low-frequency structural information of the first video, thereby better maintaining the consistency of the target part generation result with the first video in the overall style during the subsequent layer-by-layer denoising process, and ensuring the consistency of the overall style of the second video.

[0117] To better understand the above method, an application embodiment of the video processing method of this application is described in detail below.

[0118] During daily shooting or video recording, while users are generally satisfied with the overall video quality, variations in the shape of specific body parts (such as a person's chest) often lead to visually disproportionate or less-than-ideal shapes, affecting the aesthetics and expressiveness of the subject. Furthermore, in scenarios with multiple subjects or multiple camera angles, the diverse body shapes make it difficult to achieve visual optimization of specific areas using a uniform approach, impacting the overall harmony of the image. These issues are prevalent in practical applications, and users' demand for corrections to these visual defects is growing.

[0119] Due to its high-quality generation capabilities and rich generative diversity, diffusion models have received widespread attention in recent years. With the development of deep learning technology, the application of diffusion models in computer vision has matured and is widely used to handle various visual tasks, including complex scenarios such as video content generation and editing. Specifically, diffusion models can not only generate semantically accurate video content based on input text prompts, but also perform fine-grained content editing of people or scenes in videos by combining text or reference video clips. In video editing tasks, large-scale models based on diffusion models have demonstrated strong controllability and generalization ability. For example, text-guided video editing methods have been applied to tasks such as video content enhancement, restoration, and style transfer, maintaining temporal consistency and spatial coherence between video sequences. Furthermore, video redrawing methods are mainly used for significant modifications to video content, such as replacing human features, reshaping local structures, and changing backgrounds. The continuous development of these technologies provides a solid technical foundation for achieving high-quality, natural, and temporally consistent video content enhancement.

[0120] Current video processing methods based on diffusion models suffer from the following three major problems:

[0121] (1) It is difficult to guarantee the accurate generation of target parts in the redrawing area;

[0122] (2) It is difficult to guarantee the consistency between non-redrawn areas and the original video;

[0123] (3) It is difficult to guarantee the overall style consistency of the generated video.

[0124] To address the aforementioned issues, this embodiment leverages the powerful image generation capabilities of the diffusion model and incorporates targeted improvements to provide a video processing method based on the diffusion model. This method can stably and efficiently achieve the adjustment of target parts in videos.

[0125] The video processing method based on the diffusion model provided in this embodiment includes multiple stages such as preprocessing, feature encoding, noise initialization, context modeling, and denoising generation.

[0126] Building upon this foundation, this embodiment introduces two key modules into the video diffusion model: a context encoder and a low-rank adaptation unit. The context encoder, which can be a ControlNet, maintains consistency between non-masked regions (such as background, limbs, etc.) and the first video, thereby preventing structural distortion or content shift during the second video generation process. The low-rank adaptation unit, on the other hand, controls the generation results within the masked region (i.e., the target area), ensuring that the target area maintains consistency with the target area features in the reference image in terms of shape and style, thus achieving a personalized and controllable target area enhancement effect.

[0127] In addition, this embodiment also designs a noise initialization unit. During the process of generating the noise video (which can be called the initial noise) of the first video, the noise initialization unit introduces implicit guidance on the video content of the first video, so that the style, structure and other information of the first video can be encoded into the noise video, thereby better maintaining the consistency of the target part generation result with the first video in the overall style during the subsequent layer-by-layer denoising process.

[0128] The video processing method based on the diffusion model includes the following steps:

[0129] I. Preprocessing:

[0130] With highly mature image generation models, using reference images to generate target areas in video processing offers higher accuracy and visual consistency. By introducing specific images as guides for the shape and texture of target areas, the structural features of the target region can be more accurately reproduced, ensuring that the edited result closely matches the reference image in terms of shape and texture. This method demonstrates stronger performance in applications requiring strict adherence to a specific image or pursuing natural realism, and is particularly suitable for high-quality content generation and professional-grade virtual avatar creation.

[0131] One frame can be randomly selected from several frames included in the first video x as the target frame image. For example, the first frame of the first video x can be used as the target frame image. The target frame image can be input into a high-quality image generation model, which will then enhance the target region in the target frame image to obtain a reference image x1. This reference image x1 serves as a reference for the intensity and style of the target region generation in the diffusion model, guiding the intensity and style of the target region enhancement.

[0132] The first video x is input into the multimodal large model to obtain semantic description information x2 for the video content of the first video. This semantic description information x2 is used to enhance the semantic and style consistency control in the second video generation process.

[0133] The first video x is input into the segmentation model so that the segmentation model can segment each frame of the first video x. The target region of each frame is extracted by the segmentation algorithm, and the corresponding target region mask is generated. The target region mask is then applied to the first video x, and the non-masked regions are retained to obtain the masked video of the first video x, which is used for the control generation of the target region in the subsequent process.

[0134] The target area mask obtained from segmentation may be inaccurate. In order to better cover the target area that needs to be adjusted, the target area mask can be etched to expand the mask area.

[0135] II. Feature Encoding:

[0136] Based on the text encoder, the semantic description information is encoded, and the feature representation of the semantic description information is extracted to obtain the video text features.

[0137] The mask video and reference image x1 are encoded using a variational autoencoder (VAE), and their respective feature representations are extracted to obtain the mask video features and reference image features.

[0138] III. Noise Initialization:

[0139] In the process of generating the noisy video of the first video, an implicit guidance mechanism for the video content of the first video is introduced to encode the structure and style information of the first video into the noise, thus obtaining the noisy video of the first video.

[0140] Specifically, based on the noise addition mechanism of the denoising diffusion probability model, noise is gradually added to the first video up to a set noise intensity σ. This set noise intensity σ is lower than the pure noise intensity. This is equivalent to using the video in the middle stage of noise addition as the noise video of the first video during the process of adding noise to the first video, so that the noise video of the first video can retain some low-frequency structural information of the first video.

[0141] Among them, the noise addition mechanism based on the denoising diffusion probability model, the process of gradually adding noise to the first video is shown in Equation (4).

[0142] (4)

[0143] In the formula, This represents the video with added noise at time step t. This represents the video before noise addition at time step t. This represents the cumulative noise attenuation coefficient corresponding to time step t. Let represent the Gaussian noise obtained by random sampling, as shown in equation (5).

[0144] (5)

[0145] In the formula, Indicates a Gaussian distribution. Let represent the covariance matrix.

[0146] Providing a structurally controllable noise input to the diffusion model can improve the stylistic consistency and coherence between the generated target area and the first video, ensuring that the overall style of the generated second video is consistent with the first video.

[0147] IV. Contextual Modeling:

[0148] The reference image features, video text features, masked video features, and noisy video of the first video can be input into the context encoder of the diffusion model, so that the context encoder performs context encoding processing on the reference image features, video text features, masked video features, and noisy video of the first video, and fuses them to generate context-coded features with semantic consistency.

[0149] By introducing a context encoder, the mask video features of the mask video can be integrated into the diffusion model, realizing the separate modeling of background context and target region generation, thereby improving the structural stability and semantic consistency of the generated second video.

[0150] The design of the context encoder not only guides the deep fusion of conditional features such as video text features and reference image features with spatiotemporal features in the first video by encoding additional given conditional information, such as video text features, reference image features or other modal inputs, thereby introducing semantic guidance and content control in the second video generation process, but also enables the structural injection of features into specific regions to achieve the effect of local editing without destroying the overall consistency.

[0151] Specifically, in the diffusion model, conditional features such as video text features and reference image features are added to the feature representation of non-critical regions instead of the feature representation of critical regions. This avoids interfering with the original stable and important background or main structure in the first video, thereby better preserving the overall scene layout and visual style of the first video.

[0152] Furthermore, this selective injection mechanism helps improve the temporal coherence and spatial rationality of the generated results for the target area, preventing unnatural phenomena such as background distortion and structural misalignment during video restoration or editing. In this way, the context encoder provides flexible content control capabilities while effectively ensuring the consistency and authenticity of the output video between the unedited areas and the first video, so that the generated second video not only meets the user's specified editing intentions but also maintains a natural and smooth visual experience.

[0153] V. Denoising Generation:

[0154] The diffusion model denoising backbone network can perform layer-by-layer denoising on noisy videos, predict the noise distribution at each step, and gradually obtain the denoising result of the final layer.

[0155] Specifically, reference image features, video text features, and noisy video can be input into the denoising backbone network of the diffusion model. The denoising backbone network consists of six denoising units, each of which is a converter encoder layer adapted to the requirements of the diffusion model.

[0156] The noise distribution of this step can be predicted by the first-layer denoising unit of the denoising backbone network based on the features of the reference image, the features of the video text, and the noisy video. Then, the current noisy video can be iteratively adjusted according to the set sampling strategy and the noise distribution of this step to reduce the noise ratio of the current noisy video and obtain the first-layer denoising result. The first-layer denoising result can be fused with the context coding features to obtain the first-layer denoising fusion result.

[0157] The noise distribution of this step can be predicted by the non-first-layer denoising unit of the denoising backbone network based on the denoising fusion result of the previous layer. Then, the noise distribution of the current noisy video can be iteratively adjusted according to the set sampling strategy and the noise distribution of this step to reduce the noise ratio of the current noisy video and obtain the non-first-layer denoising processing result. The non-first-layer denoising processing result is then fused with the context coding features to obtain the non-first-layer denoising fusion result.

[0158] The input to the first-layer denoising unit consists of three main parts. The first part is the noisy video. The second part is the reference image features and video text features. The third part is a time series encoding T.

[0159] The formula for calculating the loss function of the first-layer denoising unit is shown in equation (1).

[0160] (1)

[0161] in, This represents Gaussian noise obtained through random sampling during the noise addition process. This represents the first part of the input, namely the noisy video. This represents the input for the second part, namely, the reference image features and the video text features. This represents the third part of the input, namely the encoding T of a time series. This represents a noise prediction model. This represents the Mean Squared Error loss function. Additionally, each denoising unit can be activated during the inference process.

[0162] VI. Low-rank adaptation:

[0163] Larger models outperform smaller models not only because of their more complex structure, but more importantly, because they are typically trained on massive amounts of data. Therefore, fine-tuning a pre-trained large model can achieve higher cost-effectiveness with limited device resources and data. This is one of the mainstream methods for efficient fine-tuning of large models.

[0164] The low-rank adaptation unit simplifies the originally complex dense layer parameter update process into learning the low-rank matrix by using the principle of matrix low-rank decomposition. This significantly reduces the number of parameters involved in training, making the training and inference of the entire model more lightweight, while retaining the powerful capabilities of the original model.

[0165] This embodiment fine-tunes the diffusion model using a low-rank adaptive unit. While keeping the diffusion model parameters unchanged, it enhances the diffusion model's ability to capture and reproduce the features of the target area in the reference image, ensuring that the generated area of ​​the target area is highly consistent with the reference image in detail.

[0166] Specifically, the noisy calibration features can be input into the low-rank adaptation (LoRA) unit of the diffusion model, so that the low-rank adaptation unit can obtain a lightweight correction signal, i.e., low-rank increment, by learning the features common to the target part.

[0167] The noisy calibration features can be corrected based on the low-rank increment, resulting in the corrected noisy calibration features. Based on the corrected noisy calibration features, the noise distribution of this step can be predicted. Then, based on the set sampling strategy and the noise distribution of this step, the current noisy video is iteratively adjusted to reduce the noise ratio of the current noisy video, thus obtaining the first-layer denoising result.

[0168] Furthermore, during the process of obtaining the non-first-layer denoising results, the previous-layer denoising fusion results can be input into the low-rank adaptation unit of the diffusion model, so that the low-rank adaptation unit can obtain a lightweight correction signal, i.e., low-rank increment, by learning the common features of the target part.

[0169] The previous layer's denoising and fusion result can be corrected based on the low-rank increment, resulting in a corrected previous layer denoising and fusion result. Based on this corrected result, the noise distribution for this step can be predicted. Then, based on the set sampling strategy and the noise distribution for this step, the current noisy video can be iteratively adjusted to reduce the noise ratio, thus obtaining the non-first layer denoising processing result.

[0170] Low-rank adaptive unit structure such as Figure 3 As shown in the diagram. Here, h represents the input to the low-rank adaptation unit, W represents the pre-trained weight parameters, and A and B represent the trainable model parameters used for fine-tuning. h1 represents the scaling factor. h1 represents the output result of the original pre-trained weights after fine-tuning, which can be expressed as equation (2):

[0171] (2)

[0172] This embodiment further designs the low-rank adaptation unit so that the generated features are applied to the target region and not to non-target regions. The loss function formula during training is shown in equation (3).

[0173] (3)

[0174] in, This represents the loss function calculated when training low-rank adaptive units. express The original features of the masked region at any given time, i.e., the original features of the target region. This represents Gaussian noise obtained through random sampling during the noise addition process. Indicates noisy calibration features. The encoding T represents a time series. This represents the noise prediction model. The loss function ensures that the target region in the generated second video maintains consistency with the target region features in the reference image.

[0175] The overall structure diagram of the video processing method based on the diffusion model is as follows: Figure 4 As shown in the figure, the diffusion model includes a text encoder, a variational encoder, a noise initialization unit, a context encoder, a denoising backbone network, and a low-rank adaptation unit.

[0176] First, the pre-trained weights of each unit of the diffusion model are loaded. Then, the first video is fed into the image generation model and the multimodal model to generate an enhanced reference image and semantic description information of the video content of the first video. Next, the first video is processed using a segmentation model to generate a corresponding target region mask, and this target region mask is applied to the first video to obtain a masked video.

[0177] Then, the first video is sent to the noise initialization unit. During the process of generating the noise video of the first video, an implicit guidance mechanism for the video content of the first video is introduced to encode the structure and style information of the first video into the noise, thus obtaining the noise video of the first video.

[0178] Next, the semantic description information is encoded using a text encoder, and feature representations of the semantic description information are extracted to obtain video text features. Then, the masked video and reference image are encoded using a variational autoencoder, and their respective feature representations are extracted to obtain masked video features and reference image features.

[0179] The reference image features, video text features, masked video features, and noisy video of the first video can be input into two denoising units in the context encoder of the diffusion model. These two denoising units then perform context encoding on the reference image features, video text features, masked video features, and noisy video of the first video, respectively, and fuse them to generate semantically consistent context-coded features. The outputs of the two denoising units are identical. The output of one denoising unit is input into the first three layers of the denoising backbone network, and the output of the other denoising unit is input into the last three layers of the denoising backbone network.

[0180] The denoising backbone network of the diffusion model can input reference image features, video text features, and noisy video into the model. The denoising backbone network consists of six denoising units, each of which is a converter encoder layer adapted to the requirements of the diffusion model. During the process of the denoising backbone network performing layer-by-layer denoising processing on the reference image features, video text features, and noisy video based on the multiple denoising units (which can be called the layer-by-layer forward propagation process), the denoising results of each layer are fused with the context-coded features to achieve condition-guided content generation.

[0181] Specifically, noisy videos can be feature-processed to obtain noisy features; based on reference image features and video text features, the noisy features are conditionally calibrated to obtain noisy calibration features. These noisy calibration features can be input into the low-rank adaptation (LoRA) unit of the diffusion model, allowing the LoRA unit to learn features common to the target region and obtain a lightweight correction signal, i.e., a low-rank increment. The noisy calibration features can then be corrected based on the low-rank increment to obtain the corrected noisy calibration features.

[0182] The noise distribution of this step can be predicted by the first-layer denoising unit of the denoising backbone network based on the corrected noisy calibration features. Then, the current noisy video is iteratively adjusted according to the set sampling strategy and the noise distribution of this step to reduce the noise ratio of the current noisy video, thus obtaining the first-layer denoising result. The first-layer denoising result can be fused with the context coding features to obtain the first-layer denoising fusion result.

[0183] The denoising and fusion result of the previous layer can be input into the low-rank adaptation unit of the diffusion model, so that the low-rank adaptation unit can learn the common features of the target parts and obtain a lightweight correction signal, i.e., a low-rank increment. The denoising and fusion result of the previous layer can be corrected based on the low-rank increment to obtain the corrected denoising and fusion result of the previous layer.

[0184] The noise distribution of this step can be predicted by the non-first-layer denoising units of the denoising backbone network based on the corrected denoising fusion result of the previous layer. Then, the current noisy video is iteratively adjusted according to the set sampling strategy and the noise distribution of this step to reduce the noise ratio of the current noisy video, thus obtaining the non-first-layer denoising result. The non-first-layer denoising result is then fused with the context coding features to obtain the non-first-layer denoising fusion result.

[0185] Finally, the denoising result of the last layer is decoded to obtain the second video after adjusting the target area in the first video. The entire process achieves high-quality editing of local areas of the first video while maintaining the coherence and naturalness of the overall content.

[0186] The first video can be either the entire video or a video segment. When the first video is the entire video, the amount of data in the first video is generally small; when the first video is a video segment, the second video, after adjusting the target part in the first video, is also a video segment. After obtaining the second video, it is spliced ​​back into the corresponding position in the original video to obtain the final complete output video.

[0187] This embodiment is a video processing method guided by the first frame, so a diffusion model can be trained to generate target parts following the first frame.

[0188] This embodiment can address, for example... Figure 4 In the overall architecture diagram shown, the context encoder and low-rank adaptation unit are trained independently, eliminating the need to train other units. Specifically, the context encoder can be trained first, followed by freezing it and training the low-rank adaptation unit. The loss function can be implemented by adding the global mean squared error loss function to the mask region mean squared error loss function.

[0189] The video processing method based on the diffusion model provided in this embodiment not only achieves natural target part adjustment effects but also provides the ability to generate target part adjustment effects similar to reference images using reference images. Specifically, leveraging the powerful generation capabilities of the diffusion model, the redrawing model algorithm can generate target parts of similar size based on reference images, providing higher accuracy and visual consistency, thus ensuring the consistency of the overall style of the second video. Secondly, by combining a context encoder, contextual features are injected into non-target part regions to preserve the background, thereby better preserving the overall scene layout and visual style of the first video and ensuring the consistency between the non-redrawing regions and the first video, thus improving the structural stability and semantic consistency of the second video. Furthermore, considering the anatomical structure inherent in the target object itself, a low-rank adaptive unit is designed to learn the common features of the target parts, making the generated target parts more natural and ensuring that the redrawing region accurately generates the target parts. Finally, to maintain stylistic consistency, a noise initialization module was introduced during the inference phase. This module implicitly encodes information from the first video into the noisy video, allowing the noisy video to retain some low-frequency structural information from the first video. This ensures that the generated target part is more consistent with the first video in overall style during subsequent layer-by-layer denoising, thus guaranteeing the overall stylistic consistency of the second video.

[0190] In summary, this embodiment proposes a stable, controllable, and style-consistent video processing method by combining multiple techniques such as diffusion model, low-rank adaptive unit, context modeling, and noise initialization, which has good practicality and promotion value.

[0191] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0192] Based on the same inventive concept, this application also provides a video processing apparatus for implementing the video processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more video processing apparatus embodiments provided below can be found in the limitations of the video processing method described above, and will not be repeated here.

[0193] In one exemplary embodiment, such as Figure 5 As shown, a video processing apparatus is provided, wherein:

[0194] The reference image acquisition module 501 is used to determine a reference image from a plurality of frames of images included in the first video; the first video includes a plurality of frames of images, each frame of images including a target part;

[0195] The video text feature acquisition module 502 is used to obtain video text features based on the semantic description information of the video content of the first video;

[0196] The mask video acquisition module 503 is used to obtain the mask video of the first video based on the mask processing for the target area;

[0197] The context coding feature acquisition module 504 is used to perform context coding processing on the reference image features of the reference image, the video text features, the mask video features of the mask video, and the noisy video of the first video to obtain context coding features;

[0198] The fusion module 505 is used to fuse the denoising result of each layer with the context coding features during the layer-by-layer denoising process based on the reference image features, the video text features and the noisy video.

[0199] The second video acquisition module 506 is used to obtain a second video after adjusting the target part based on the denoising processing result of the last layer.

[0200] In one embodiment, the video text feature acquisition module 502 is further configured to: input the first video into a multimodal large model to obtain semantic description information of the video content of the first video; and extract features from the semantic description information through a text encoder of a diffusion model to obtain video text features.

[0201] In one embodiment, the fusion module 505 is further configured to: input the reference image features, the video text features, and the noisy video into the denoising backbone network of the diffusion model; wherein the denoising backbone network is configured to: perform denoising processing on the reference image features, the video text features, and the noisy video through the first-layer denoising unit of the denoising backbone network to obtain a first-layer denoising processing result; fuse the first-layer denoising processing result with the context coding features to obtain a first-layer denoising fusion result; perform denoising processing on the previous-layer denoising fusion result through the non-first-layer denoising unit of the denoising backbone network to obtain a non-first-layer denoising processing result; and fuse the non-first-layer denoising processing result with the context coding features to obtain a non-first-layer denoising fusion result.

[0202] In one embodiment, the fusion module 505 is further configured to: perform feature processing on the noisy video to obtain noisy features; perform conditional calibration on the noisy features based on the reference image features and the video text features to obtain noisy calibration features; and obtain the first-layer denoising processing result based on the noisy calibration features.

[0203] In one embodiment, the fusion module 505 is further configured to: input the noisy calibration features into the low-rank adaptation unit of the diffusion model to obtain a low-rank increment; correct the noisy calibration features according to the low-rank increment to obtain corrected noisy calibration features; and obtain the first-layer denoising result according to the corrected noisy calibration features.

[0204] In one embodiment, the reference image acquisition module 501 is further configured to: select a target frame image from a plurality of frame images included in the first video; input the target frame image into an image generation model, so that the image generation model enhances the target part in the target frame image to obtain a reference image.

[0205] In one embodiment, the apparatus further includes a noise video acquisition module, configured to: acquire a set noise intensity; the set noise intensity is lower than the pure noise intensity; and gradually add noise to the first video up to the set noise intensity to obtain a noise video of the first video.

[0206] Each module in the aforementioned video processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0207] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database stores data for embodiments of the video processing method. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a video processing method.

[0208] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0209] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0210] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0211] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0212] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0213] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0214] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0215] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A video processing method, characterized in that, The method includes: A reference image is determined from a plurality of frames included in the first video; the first video includes a plurality of frames, each frame including the target region; Based on the semantic description information of the video content of the first video, video text features are obtained; Based on the masking process applied to the target area, the masked video of the first video is obtained; Context encoding is performed on the reference image features of the reference image, the video text features, the mask video features of the mask video, and the noisy video of the first video to obtain context encoded features; In the process of performing layer-by-layer denoising based on reference image features, video text features, and noisy video, the denoising result of each layer is fused with the context coding features; Based on the denoising results of the last layer, a second video is obtained after adjusting the target area.

2. The method according to claim 1, characterized in that, The step of obtaining video text features based on semantic description information of the video content of the first video includes: Input the first video into the multimodal large model to obtain semantic description information of the video content of the first video; The semantic description information is extracted using a text encoder based on a diffusion model to obtain video text features.

3. The method according to claim 1, characterized in that, In the process of performing layer-by-layer denoising based on reference image features, video text features, and noisy video, the denoising result of each layer is fused with the context coding features, including: The denoising backbone network of the reference image features, the video text features, and the noise video input diffusion model; The denoising backbone network is configured to: process the reference image features, the video text features, and the noisy video through its first-layer denoising unit to obtain a first-layer denoising result; fuse the first-layer denoising result with the context coding features to obtain a first-layer denoising fusion result; process the previous-layer denoising fusion result through the non-first-layer denoising unit of the denoising backbone network to obtain a non-first-layer denoising result; and fuse the non-first-layer denoising result with the context coding features to obtain a non-first-layer denoising fusion result.

4. The method according to claim 3, characterized in that, The denoising process performed on the reference image features, the video text features, and the noisy video to obtain the first-layer denoising result includes: Noisy video is subjected to feature processing to obtain noisy features; Based on the reference image features and the video text features, the noisy features are conditionally calibrated to obtain the noisy calibrated features; Based on the noisy calibration features, the first-layer denoising result is obtained.

5. The method according to claim 4, characterized in that, The step of obtaining the first-layer denoising result based on the noisy calibration features includes: The noisy calibration features are input into the low-rank adaptation unit of the diffusion model to obtain the low-rank increment. Based on the low-rank increment, the noisy calibration feature is corrected to obtain the corrected noisy calibration feature. Based on the corrected noisy calibration features, the first-layer denoising result is obtained.

6. The method according to claim 1, characterized in that, Determining the reference image from the plurality of frames included in the first video includes: Select the target frame image from the several frames included in the first video; The target frame image is input to an image generation model, which enhances the target region in the target frame image to obtain a reference image.

7. The method according to claim 1, characterized in that, Obtaining the noisy video of the first video includes: Obtain a set noise intensity; the set noise intensity is lower than the pure noise intensity. Noise is gradually added to the first video until a set noise level is reached, resulting in a noisy video of the first video.

8. A video processing apparatus, characterized in that, The device includes: A reference image acquisition module is used to determine a reference image from several frames of images included in the first video; the first video includes several frames of images, and each frame of images includes a target part. The video text feature acquisition module is used to obtain video text features based on the semantic description information of the video content of the first video; The mask video acquisition module is used to obtain the mask video of the first video based on the mask processing for the target area; The context coding feature acquisition module is used to perform context coding processing on the reference image features of the reference image, the video text features, the mask video features of the mask video, and the noisy video of the first video to obtain context coding features; The fusion module is used to fuse the denoising result of each layer with the context coding features during the layer-by-layer denoising process based on reference image features, the video text features, and noisy video. The second video acquisition module is used to obtain a second video after adjusting the target part based on the denoising processing result of the last layer.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images