A video processing method based on a multi-condition control diffusion model and related equipment

Abstract

The application discloses a video processing method based on a multi-condition control diffusion model and related equipment, and the method comprises the following steps: adopting a double-criterion detection strategy to perform frame extraction processing on an original video to obtain a target key frame sequence; performing segmentation processing on the original video according to the target key frame sequence to obtain a plurality of original video segments; performing static feature extraction and compression on each original video segment to obtain static condition data; performing dynamic feature extraction and compression on each original video segment to obtain dynamic condition data; performing data preprocessing on the dynamic condition data to obtain a dynamic condition noise tensor; inputting the static condition data and the dynamic condition noise tensor into a multi-condition control diffusion model to perform condition diffusion reconstruction processing, and outputting a plurality of reconstructed video segments. The application can realize fine regulation and control of perceptual quality and compression ratio, improve perceptual quality and compression efficiency, and improve the detail restoration capability, and can be widely applied to the technical field of data processing.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a video processing method and related equipment based on a multi-condition controlled diffusion model. Background Technology

[0002] Current video compression technologies are mainly divided into two categories: traditional coding standards and neural compression methods based on deep learning. However, both have significant limitations: traditional methods prioritize distortion metrics at the expense of perceptual quality, while neural methods focus on bitrate control but neglect detail generation. In addition, generative models such as Generative Adversarial Networks (GANs) can improve visual fidelity, but they are prone to mode collapse or artifacts, making it difficult to stably balance compression rate and reconstruction quality, thus limiting the reconstruction quality of dynamic scenes.

[0003] In recent years, diffusion models have shown the potential to generate high-fidelity results in image compression, but their application in the video field still faces three major challenges: First, directly extending the image diffusion model will lead to poor temporal consistency, manifested as inter-frame flickering or motion breakage; second, the iterative generation mechanism causes high decoding latency, making it difficult to put into practical use; and third, using a single condition to control the diffusion process leads to insufficient reconstruction fidelity.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The embodiments of this application aim to at least partially solve one of the technical problems in the related art. Therefore, the main objective of the embodiments of this application is to propose a video processing method and related equipment based on a multi-condition controlled diffusion model, which can achieve fine-tuning of perceived quality and compression ratio, improve detail restoration capabilities, and thus enhance perceived quality and compression efficiency.

[0006] To achieve the above objectives, one aspect of this application proposes a video processing method based on a multi-condition controlled diffusion model, the method comprising the following steps:

[0007] The original video is acquired, and a dual-criteria detection strategy is used to extract frames from the original video to obtain the target keyframe sequence.

[0008] A segmentation algorithm is used to segment the original video based on the target keyframe sequence to obtain several original video segments;

[0009] Static feature extraction and compression are performed on each of the original video segments to obtain static conditional data; wherein, the static conditional data includes keyframe compressed data and text description data;

[0010] Dynamic features are extracted and compressed from each of the original video segments to obtain dynamic conditional data; wherein, the dynamic conditional data includes segmentation sequences, human motion data, and optical flow field data;

[0011] The dynamic condition data is preprocessed to obtain the dynamic condition noise tensor;

[0012] The static condition data and the dynamic condition noise tensor are input into a multi-condition controlled diffusion model for conditional diffusion reconstruction processing, outputting several reconstructed video segments, and a reconstructed video is generated based on the several reconstructed video segments.

[0013] In some embodiments, the dual-criteria detection strategy includes a shot boundary detection method and a fixed-interval sampling method. The step of acquiring the original video and performing frame extraction processing on the original video using the dual-criteria detection strategy to obtain a target keyframe sequence includes:

[0014] Obtain the original video;

[0015] Using the aforementioned shot boundary detection method, the inter-frame shot switching probability in the original video is calculated, and the first keyframe sequence is determined based on the inter-frame shot switching probability and a preset switching probability threshold.

[0016] Using the fixed-interval sampling method, keyframes are marked in the original video according to a preset frame interval to obtain a second keyframe sequence;

[0017] The target keyframe sequence is constructed based on the first keyframe sequence and the second keyframe sequence.

[0018] In some embodiments, the step of extracting and compressing static features from each of the original video segments to obtain static conditional data includes:

[0019] Frame feature extraction is performed on each of the original video segments to obtain the first keyframe and the last keyframe of each of the original video segments.

[0020] The first and last keyframes in each of the original video segments are compressed using an image encoder to obtain the keyframe compressed data.

[0021] The original video segments are processed by a multimodal model to extract text, thereby obtaining the text description data.

[0022] The static condition data is constructed based on the keyframe compressed data and the text description data.

[0023] In some embodiments, the step of extracting and compressing dynamic features from each of the original video segments to obtain dynamic conditional data includes:

[0024] Panoramic segmentation is performed on the video frames in each of the original video segments to obtain a panoramic segmentation map of each video frame, and the object contours of the panoramic segmentation map of each video frame are extracted.

[0025] The segmentation sequence is obtained by fitting the contour of each object using a Bézier curve;

[0026] Extract the three-dimensional joint sequence from each of the original video segments, and project the three-dimensional joint sequence from each of the original video segments into two-dimensional coordinate data;

[0027] Based on a preset motion area threshold, the two-dimensional coordinate data in each of the original video segments are filtered to obtain the human motion data;

[0028] Calculate the dense optical flow in each of the original video segments to obtain the optical flow field data;

[0029] The dynamic condition data is constructed based on the segmentation sequence, the human motion data, and the optical flow field data.

[0030] In some embodiments, the step of preprocessing the dynamic condition data to obtain a dynamic condition noise tensor includes:

[0031] The dynamic conditional data is converted into visual modal data, and the visual modal data is input into a pre-trained encoder for encoding processing to obtain initial low-dimensional latent variables;

[0032] A dynamic condition control strategy is adopted to transform the initial low-dimensional latent variable into a target low-dimensional latent variable, and then concatenate the target low-dimensional latent variable with noise to obtain the dynamic condition noise tensor. The dynamic condition control strategy includes a random drop mechanism and a modal embedding mechanism. The modal embedding mechanism is used to distinguish the role states of the condition, and the role states include normal conditions and dropout states.

[0033] In some embodiments, the step of converting the dynamic conditional data into visual modal data and inputting the visual modal data into a pre-trained encoder for encoding processing to obtain initial low-dimensional latent variables includes:

[0034] The object contours in the segmented sequence after Bézier curve fitting are rendered as a binary mask sequence;

[0035] Connect the two-dimensional joints in the human motion data to generate a skeleton sequence, and visualize the skeleton sequence as a motion trajectory diagram that evolves over time.

[0036] The motion vectors in the optical flow field data are mapped to an RGB image sequence using a color encoding method;

[0037] The visual modality data is constructed based on the binary mask sequence, the motion trajectory map, and the RGB image sequence.

[0038] The visual modality data is input into the pre-trained encoder for encoding processing to obtain the initial low-dimensional latent variables.

[0039] To achieve the above objectives, another aspect of this application proposes a video processing apparatus based on a multi-condition controlled diffusion model, the apparatus comprising the following modules:

[0040] The keyframe sequence extraction module is used to acquire the original video and perform frame extraction processing on the original video using a dual-criteria detection strategy to obtain the target keyframe sequence.

[0041] The video segmentation module is used to segment the original video according to the target keyframe sequence using a segmentation algorithm to obtain several original video segments;

[0042] The static condition data acquisition module is used to extract and compress static features from each of the original video segments to obtain static condition data; wherein, the static condition data includes keyframe compressed data and text description data;

[0043] The dynamic condition data acquisition module is used to extract and compress dynamic features from each of the original video segments to obtain dynamic condition data; wherein, the dynamic condition data includes segmentation sequences, human motion data, and optical flow field data;

[0044] The conditional noise tensor acquisition module is used to preprocess the dynamic conditional data to obtain the dynamic conditional noise tensor.

[0045] The conditional diffusion reconstruction module is used to input the static conditional data and the dynamic conditional noise tensor into a multi-conditional controlled diffusion model for conditional diffusion reconstruction processing, output several reconstructed video segments, and generate a reconstructed video based on the several reconstructed video segments.

[0046] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0047] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0048] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0049] The embodiments of this application include at least the following beneficial effects: This application provides a video processing method and related equipment based on a multi-condition controlled diffusion model. This method acquires the original video and performs frame extraction processing on the original video using a dual-criteria detection strategy to obtain a target keyframe sequence; it then uses a segmentation algorithm to segment the original video according to the target keyframe sequence to obtain several original video segments; it performs static feature extraction and compression on each original video segment to obtain static conditional data; wherein, the static conditional data includes keyframe compressed data and text description data; it then performs dynamic feature extraction and compression on each original video segment to obtain dynamic conditional data; wherein, the dynamic conditional data includes segmentation sequences, human motion data, and optical flow field data; it performs data preprocessing on the dynamic conditional data to obtain a dynamic conditional noise tensor; it inputs the static conditional data and the dynamic conditional noise tensor into a multi-condition controlled diffusion model for conditional diffusion reconstruction processing, outputting several reconstructed video segments, and generating a reconstructed video based on these reconstructed video segments. This application's embodiments leverage the high-fidelity generation capabilities of a multi-condition controlled diffusion model. At low bitrates, it avoids issues like blockiness and blurring common in traditional methods, generating video content more consistent with human visual perception. By combining static and dynamic condition control, it explicitly models spatiotemporal semantics during compression, enabling fine-tuning of perceptual quality and compression ratio, enhancing detail reproduction, and thus improving both perceptual quality and compression efficiency. Simultaneously, by embedding static and dynamic condition data into the diffusion process, static conditions enhance the visual accuracy and semantic consistency of static scenes, while dynamic conditions enhance the temporal consistency of dynamic scenes. These two factors work together to ensure visual and semantic accuracy of reconstructed frames, while maintaining inter-frame coherence, avoiding flickering or motion breaks, and accurately restoring the semantic and motion features of the video. Furthermore, the multi-condition controlled diffusion model supports flexible configuration of dynamically adjusted compression ratios and perceptual weights, thereby improving adaptability to different application scenarios. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating the steps of a video processing method based on a multi-condition controlled diffusion model provided in an embodiment of this application.

[0051] Figure 2 yes Figure 1 The flowchart of step S101 in the text;

[0052] Figure 3 yes Figure 1 The flowchart of step S103 in the text;

[0053] Figure 4 yes Figure 1 The flowchart of step S104 in the text;

[0054] Figure 5 This is a schematic diagram illustrating the compressed representation of dynamic conditions under different thresholds provided in the embodiments of this application;

[0055] Figure 6 yes Figure 1 The flowchart of step S105 in the text;

[0056] Figure 7 This is a schematic diagram of the decoding end conditional diffusion reconstruction process provided in the embodiments of this application;

[0057] Figure 8 This is a flowchart illustrating a video processing method based on a multi-condition controlled diffusion model provided in an embodiment of this application;

[0058] Figure 9 This is a schematic diagram of the structure of a video processing device based on a multi-condition control diffusion model provided in an embodiment of this application;

[0059] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0061] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0062] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0064] Current video compression technologies are mainly divided into two categories: traditional coding standards and neural compression methods based on deep learning. However, both have significant limitations. Traditional video coding standards (such as H.264 / H.265 / VVC) are based on a hybrid coding framework. Although they have high compatibility and real-time performance, their manually designed modules limit the optimization space. Improving efficiency requires traversing a large number of coding modes, resulting in high computational complexity. Especially in low bitrate scenarios, these methods are prone to block artifacts, blurring, and color distortion, making it difficult to meet high-fidelity requirements.

[0065] To overcome the limitations of traditional methods, neural video compression techniques (such as DVC and HiFiC) have emerged. These methods learn spatiotemporal redundancy representations through end-to-end trained neural networks, and their performance has surpassed traditional coding in some scenarios. However, their optimization targets often rely on pixel-level metrics (such as PSNR / SSIM), resulting in overly smooth and detail-lacking generated images that fail to maintain perceptual quality. Furthermore, while generative models such as Generative Adversarial Networks (GANs) can improve visual fidelity, they are prone to pattern collapse or artifacts, making it difficult to stably balance compression ratio and reconstruction quality.

[0066] In recent years, diffusion models have shown the potential to generate high-fidelity results in image compression, but their application in the video field still faces three major challenges: First, directly extending the image diffusion model can lead to poor temporal consistency, manifested as inter-frame flickering or motion breaks; second, the iterative generation mechanism results in high decoding latency, making it difficult to put into practical use; and third, the use of single-condition control leads to insufficient reconstruction fidelity.

[0067] In summary, none of the relevant technologies can simultaneously optimize the three objectives of bitrate, distortion, and perception: traditional methods prioritize distortion metrics at the expense of perceptual quality, neural methods focus on bitrate control but neglect detail generation, and generative models, while improving visual effects, struggle to balance efficiency and stability, resulting in limited quality in dynamic scene reconstruction.

[0068] In view of this, this application provides a video processing method and related device based on a multi-condition controlled diffusion model. This method acquires the original video and performs frame extraction processing on the original video using a dual-criteria detection strategy to obtain a target keyframe sequence. A segmentation algorithm is used to segment the original video according to the target keyframe sequence, resulting in several original video segments. Static feature extraction and compression are performed on each original video segment to obtain static conditional data, which includes keyframe compression data and text description data. Dynamic feature extraction and compression are performed on each original video segment to obtain dynamic conditional data, which includes segmentation sequences, human motion data, and optical flow field data. Data preprocessing is performed on the dynamic conditional data to obtain a dynamic conditional noise tensor. The static conditional data and the dynamic conditional noise tensor are input into a multi-condition controlled diffusion model for conditional diffusion reconstruction processing, outputting several reconstructed video segments, and a reconstructed video is generated based on these reconstructed video segments. This application's embodiments leverage the high-fidelity generation capabilities of a multi-condition controlled diffusion model. At low bitrates, it avoids issues like blockiness and blurring common in traditional methods, generating video content more consistent with human visual perception. By combining static and dynamic condition control, it explicitly models spatiotemporal semantics during compression, enabling fine-tuning of perceptual quality and compression ratio, enhancing detail reproduction, and thus improving both perceptual quality and compression efficiency. Simultaneously, by embedding static and dynamic condition data into the diffusion process, static conditions enhance the visual accuracy and semantic consistency of static scenes, while dynamic conditions enhance the temporal consistency of dynamic scenes. These two factors work together to ensure visual and semantic accuracy of reconstructed frames, while maintaining inter-frame coherence, avoiding flickering or motion breaks, and accurately restoring the semantic and motion features of the video. Furthermore, the multi-condition controlled diffusion model supports flexible configuration of dynamically adjusted compression ratios and perceptual weights, thereby improving adaptability to different application scenarios.

[0069] This application provides a video processing method based on a multi-condition controlled diffusion model, relating to the field of data processing technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited thereto. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing a video processing method based on a multi-condition controlled diffusion model, but is not limited to the above forms.

[0070] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0071] Please see Figure 1 , Figure 1 This is an optional flowchart of a video processing method based on a multi-condition controlled diffusion model provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0072] Step S101: Obtain the original video and perform frame extraction processing on the original video using a dual-criteria detection strategy to obtain the target keyframe sequence;

[0073] Please see Figure 2 , Figure 2 yes Figure 1 The flowchart of step S101 in the text is as follows: Figure 2 As shown, in some embodiments, step S101 may include, but is not limited to, steps S201 to S204:

[0074] Step S201: Obtain the original video;

[0075] Step S202: Using the lens boundary detection method, calculate the inter-frame shot switching probability in the original video, and determine the first keyframe sequence based on the inter-frame shot switching probability and a preset switching probability threshold.

[0076] Step S203: Using the fixed interval sampling method, keyframes are marked in the original video according to the preset frame interval to obtain a second keyframe sequence;

[0077] Step S204: Construct the target keyframe sequence based on the first keyframe sequence and the second keyframe sequence.

[0078] Optionally, the dual-criteria detection strategy includes a lens boundary detection method and a fixed-interval sampling method.

[0079] Among them, the shot boundary detection method uses the TransNetV2 model to calculate the inter-frame shot switching probability. ,when A frame value greater than 0.5 is marked as a keyframe (0.5 is a preset switching probability threshold). For ease of distinction, this embodiment refers to the keyframe sequence acquired by the lens boundary detection method as the first keyframe sequence. The fixed interval sampling method refers to forcibly inserting keyframes every w frames (w is an adjustable hyperparameter) to balance compression ratio and reconstruction quality. For ease of distinction, this embodiment refers to the keyframe sequence acquired by the fixed interval sampling method as the second keyframe sequence.

[0080] The target keyframe sequence includes a first keyframe sequence acquired by the lens boundary detection method and a second keyframe sequence acquired by the fixed interval sampling method.

[0081] Step S102: Using a segmentation algorithm, the original video is segmented according to the target keyframe sequence to obtain several original video segments;

[0082] Among them, the segmentation algorithm is a method for segmenting the original video based on the keyframe sequence.

[0083] In the specific implementation, the original video is segmented based on the keyframe sequence obtained in step S101. Specifically: based on the keyframe sequence... The original video is divided into M independent segments. .

[0084] It is worth mentioning that if a keyframe is triggered by a camera cut, the starting frame of its adjacent segment is shifted back by 1 frame to avoid ambiguity in the ownership of the transition frame.

[0085] By selecting keyframes and segmenting video segments, the continuity of motion within each segment can be ensured, providing a spatiotemporally aligned framework for subsequent multi-condition feature extraction and diffusion reconstruction.

[0086] Step S103: Static feature extraction and compression are performed on each of the original video segments to obtain static conditional data; wherein, the static conditional data includes keyframe compressed data and text description data;

[0087] Please see Figure 3 , Figure 3 yes Figure 1 The flowchart of step S103 in the example is as follows: Figure 3 As shown, in some embodiments, step S103 may include, but is not limited to, steps S301 to S304:

[0088] Step S301: Perform frame feature extraction processing on each of the original video segments to obtain the first keyframe and the last keyframe in each of the original video segments.

[0089] Step S302: Compress the first keyframe and the last keyframe in each of the original video segments using an image encoder to obtain the keyframe compressed data.

[0090] Step S303: The original video segments are processed by a multimodal model to extract text and obtain the text description data.

[0091] Step S304: Construct the static condition data based on the keyframe compressed data and the text description data.

[0092] The static condition data includes keyframe compressed data and text description data.

[0093] In step S103, to achieve precise video reconstruction control, the system extracts and compresses features from both static and dynamic dimensions. The static aspect includes keyframe compression and semantic description.

[0094] For keyframe compression: Image encoders such as LIC (Low-complexity Inverse DCT-based Coding) are used to compress the first and last keyframes, while preserving visual anchor points, to obtain compressed keyframe data.

[0095] For the generation of semantic descriptions (i.e., text descriptions): text descriptions are generated through a multimodal model. It should be noted that the multimodal model can adopt relevant mature models in this technical field to generate text description data in videos, and this application embodiment does not limit this.

[0096] In step S103, the core visual content is preserved by compressing keyframes and text descriptions are introduced to provide high-level semantic information, which can improve the subsequent compression rate and provide strong prior knowledge for the reconstruction process, ensuring the accuracy and rationality of the content.

[0097] Step S104: Dynamic feature extraction and compression are performed on each of the original video segments to obtain dynamic conditional data; wherein, the dynamic conditional data includes segmentation sequence, human motion data and optical flow field data;

[0098] Please see Figure 4 , Figure 4 yes Figure 1 The flowchart of step S104 in the example is as follows: Figure 4 As shown, in some embodiments, step S104 may include, but is not limited to, steps S401 to S406:

[0099] Step S401: Perform panoramic segmentation on the video frames in each of the original video segments to obtain a panoramic segmentation map of each video frame, and extract the object contours of the panoramic segmentation map of each video frame.

[0100] Step S402: Fit each object contour using a Bézier curve to obtain the segmentation sequence;

[0101] Step S403: Extract the three-dimensional joint sequence from each of the original video segments, and project the three-dimensional joint sequence from each of the original video segments into two-dimensional coordinate data;

[0102] Step S404: Filter the two-dimensional coordinate data in each of the original video segments according to a preset motion area threshold to obtain the human motion data;

[0103] Step S405: Calculate the dense optical flow in each of the original video segments to obtain the optical flow field data;

[0104] Step S406: Construct the dynamic condition data based on the segmentation sequence, the human motion data, and the optical flow field data.

[0105] The dynamic conditional data includes segmentation sequences, human motion data, and optical flow field data.

[0106] For obtaining the segmentation sequence: First, Mask2Former is used to extract the panoramic segmentation image for each frame. Then, object contours are extracted from each panoramic segmentation image. Next, an 8th-order Bézier curve is fitted to each contour to obtain the segmentation sequence. During transmission, only the control points of the Bézier curve are transmitted. Increasing the number of curves N improves the compression ratio.

[0107] For acquiring human motion data: First, SMPL (Skinned Multi-Person Linear Model) is used to extract 3D joints (i.e., three-dimensional joints). Then, the 3D joints are projected into 2D coordinates (i.e., two-dimensional coordinates) and small-scale movements are filtered (based on a preset movement area threshold ξ). Finally, the coordinates of 21 key joints are retained to obtain human motion data.

[0108] For acquiring optical flow field data: Dense optical flow is calculated based on RAFT (Recurrent All-Pairs Field Transforms), with steps... Sparse sampling and transmission of arrow-shaped representations (direction = motion angle, length = displacement) yield optical flow field data.

[0109] In step S104, dynamic information of the video is accurately captured from multiple dimensions such as optical flow, human motion, and segmentation sequence, constructing a complete and robust motion description to ensure that the reconstructed video can accurately reproduce the original, complex and diverse motion details.

[0110] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating the compressed representation of dynamic conditions under different thresholds provided in the embodiments of this application. Figure 5 The compressed representations of three dynamic conditions (segmented sequences, human motion data, and optical flow field data) at different thresholds are shown, such as Figure 5 As shown, the examples illustrate application scenarios with three different compression requirements: high compression ratio, medium compression ratio, and low compression ratio. Specifically:

[0111] Example 1, high compression level (suitable for scenarios with extremely limited bandwidth), i.e. Figure 5 The three compression representations corresponding to Level 1 have the following key parameter configurations:

[0112] A. Keyframes are sampled at a fixed interval, w=81;

[0113] B. Number of Bézier curves N=10: 8th order Bézier curves are fitted to the contours of each frame of panoramic segmentation image, and only the control parameters corresponding to the 10 longest contours are retained to maximize compression efficiency while ensuring the topology of the core object.

[0114] C. Optical flow downsampling ratio l = 128: A large step sampling strategy is adopted, and the transmission is represented by arrows (direction represents the motion angle, and length represents the displacement), which greatly reduces the amount of optical flow data;

[0115] D. Human pose proportion threshold: Set the height of the bounding rectangle of the human pose to be greater than 1 / 5 of the video height, retain only the human poses that account for a large proportion in the image, filter out small-scale or distant human figures, and reduce redundant motion data.

[0116] Example 2, medium compression level (suitable for mainstream network environments), i.e. Figure 5 The three compression representations corresponding to Level 2 have the following key parameter configurations:

[0117] A. Keyframes are sampled at a fixed interval, w=81;

[0118] B. Number of Bézier curves N=20: The longest 20 contours in each frame are retained, achieving a balance between compression efficiency and contour integrity, and more accurately restoring the edge structure of objects;

[0119] C. Optical flow downsampling ratio = 96: It can capture the main motion trend without excessively increasing the amount of data transmission;

[0120] D. Human posture proportion threshold: The height of the bounding rectangle of the human posture is greater than 1 / 8 of the video height, so as to retain more medium-scale human postures and ensure the continuity of daily activities (such as walking and talking).

[0121] Example 3, low compression level (suitable for high-quality video transmission), i.e. Figure 5 The three compression representations corresponding to Level 3 have the following key parameter configurations:

[0122] A. Keyframes are sampled at a fixed interval w=45;

[0123] B. Number of Bézier curves N=30: Retain the longest 30 contours in each frame to finely restore the edge details of objects (such as texture boundaries and small structures) and improve the sharpness of the image;

[0124] C. Optical flow downsampling ratio = 64: Optical flow vectors are sampled at 64×64 pixel block intervals to capture subtle movements (such as changes in facial expressions and slight displacement of objects) and enhance motion smoothness;

[0125] D. Human posture proportion threshold: The height of the bounding rectangle of the human posture is greater than 1 / 10 of the video height, so as to retain more small-scale human postures and ensure the accurate reproduction of complex movements (such as dance and sports competitions).

[0126] Step S105: Perform data preprocessing on the dynamic condition data to obtain the dynamic condition noise tensor;

[0127] Please see Figure 6 , Figure 6 yes Figure 1 The flowchart of step S105 in the example is as follows: Figure 6 As shown, in some embodiments, step S105 may include, but is not limited to, steps S601 to S602:

[0128] Step S601: Convert the dynamic conditional data into visual modal data, and input the visual modal data into a pre-trained encoder for encoding processing to obtain initial low-dimensional latent variables;

[0129] Step S602: A dynamic condition control strategy is adopted to convert the initial low-dimensional latent variable into a target low-dimensional latent variable, and then concatenate the target low-dimensional latent variable with noise to obtain the dynamic condition noise tensor; wherein, the dynamic condition control strategy includes a random discarding mechanism and a modal embedding mechanism, the modal embedding mechanism is used to distinguish the role state of the condition, and the role state includes normal condition and discard state.

[0130] In some specific embodiments, the step of converting dynamic conditional data into visual modal data and inputting the visual modal data into a pre-trained encoder for encoding processing to obtain initial low-dimensional latent variables may include: rendering the object contours in the segmentation sequence after Bézier curve fitting into a binary mask sequence; connecting the two-dimensional joints in the human motion data to generate a skeleton sequence and visualizing the skeleton sequence as a motion trajectory map that evolves over time; mapping the motion vectors in the optical flow field data into an RGB image sequence using a color encoding method; constructing visual modal data based on the binary mask sequence, the motion trajectory map, and the RGB image sequence; and inputting the visual modal data into a pre-trained encoder for encoding processing to obtain initial low-dimensional latent variables.

[0131] The dynamic condition control strategy includes a random drop mechanism and a modal embedding mechanism. The modal embedding mechanism is used to distinguish the role states of conditions, which include normal conditions and drop states.

[0132] Please see Figure 7 , Figure 7 This is a schematic diagram of the decoding end conditional diffusion reconstruction process provided in the embodiments of this application, such as... Figure 7 As shown, Figure 7 This includes the process of preprocessing dynamic conditional data to obtain the dynamic conditional noise tensor. In the specific implementation, firstly, dynamic conditions such as optical flow field, segmentation sequence, and human motion are converted into visual modalities. Specifically, the optical flow field maps motion vectors to an RGB image sequence through color encoding; the segmentation sequence renders the contours fitted by Bézier curves into a binary mask sequence; and human motion forms a skeleton sequence by connecting 2D joints and is visualized as a motion trajectory graph that evolves over time. Subsequently, these visual modal data—optical flow field, segmentation sequence, and human motion—are input into a pre-trained 3D-VAE encoder. Visual modal data, such as optical flow, segmentation sequences, and human motion, are encoded as initial low-dimensional latent variables. , {Optical flow field, segmentation sequence, human motion}, among which... Figure 7 In this context, X1 represents the initial low-dimensional latent variable corresponding to the optical flow field, X2 represents the initial low-dimensional latent variable corresponding to human motion, and X3 represents the initial low-dimensional latent variable corresponding to the segmentation sequence. Next, a dynamic conditional control strategy is used to first determine the target latent variable. and target latent variables splicing with noise (sponging operation as follows) Figure 7 After inputting the "C" operation module into the diffusion model, specifically, on the one hand, a random discard mechanism is applied to each condition: a certain condition is randomly masked with a probability of 0.3 and replaced with a zero tensor, whose visual modality representation is a sequence of pure black images, to prevent overfitting; on the other hand, a modality embedding vector is introduced. Distinguishing role states based on conditions: Adding modal embedding vectors Distinguishing normal conditions ( ) and discarded state ( Specifically, first, the modality embedding vector is... With initial low-dimensional latent variables Adding them together yields the target low-dimensional latent variable that can be used as the final input. : = + , Optical flow field, segmentation sequence, and human motion; then, the three target low-dimensional latent variables corresponding to these three modes are... , {Optical flow, segmentation sequence, human motion} and noise are concatenated along the feature channel dimension to obtain a dynamic conditional noise tensor, which is then used as the model input.

[0133] The initial low-dimensional latent variables include those corresponding to the optical flow field, human motion, and segmentation sequence. The target low-dimensional latent variable is obtained by concatenating the initial low-dimensional latent variables with the modal embedding vector.

[0134] The dynamic conditional noise tensor is the data to be input into the model obtained by concatenating the target low-dimensional latent variable with the noise.

[0135] In step S105, the extracted dynamic features are transformed into a dynamic conditional noise tensor that can be understood by the diffusion model, serving as a precise guide for the generation process, thereby achieving pixel-level dynamic control of the video generation process and improving the accuracy of the reconstruction action.

[0136] Step S106: Input the static condition data and the dynamic condition noise tensor into the multi-condition controlled diffusion model for conditional diffusion reconstruction processing, output several reconstructed video segments, and generate a reconstructed video based on the several reconstructed video segments.

[0137] Among them, the multi-condition control diffusion model is a video generation framework that introduces a variety of control signals (static condition data and dynamic condition data) on the basis of the basic diffusion model to achieve precise and fine-grained control over the generated video content.

[0138] like Figure 7 As shown, the diffusion process is based on the pre-trained FL2V (first frame & last frame to video) diffusion model WAN2.1, whose original inputs include the first frame, the last frame, and text descriptions. Its framework is implemented by encoding three target low-dimensional latent variables through dynamic conditional encoding. The dynamic conditional noise tensor, obtained by concatenating optical flow, segmentation sequence, human motion, and noise along the feature channel dimension, is used as the model input. The first and last keyframes of the segment, along with text descriptions, serve as initial conditions. An iterative denoising process is performed through a noise prediction network to obtain the denoised output features. These features are then remapped to the video RGB space by a video RGB decoding network D, progressively generating intermediate frames to ultimately reconstruct the entire segment. A reconstructed video is then generated based on multiple reconstructed video segments. This process, by fusing static keyframes and dynamic conditional signals, accurately restores the semantic and motion features of the video while maintaining spatiotemporal coherence.

[0139] In step S106, by embedding and spreading the static condition data and the dynamic condition data, the static conditions can enhance the visual accuracy and semantic consistency of the static scene, and the dynamic conditions can enhance the temporal consistency of the dynamic scene. The two work together to ensure the visual and semantic accuracy of the reconstructed frame, while ensuring the coherence between frames, avoiding flickering or motion breaks, and accurately restoring the semantic and motion features of the video.

[0140] Steps S101 to S106 as illustrated in this embodiment involve: acquiring the original video and performing frame extraction processing on the original video using a dual-criteria detection strategy to obtain a target keyframe sequence; employing a segmentation algorithm to segment the original video according to the target keyframe sequence to obtain several original video segments; performing static feature extraction and compression on each original video segment to obtain static conditional data, wherein the static conditional data includes keyframe compressed data and text description data; performing dynamic feature extraction and compression on each original video segment to obtain dynamic conditional data, wherein the dynamic conditional data includes segmentation sequences, human motion data, and optical flow field data; performing data preprocessing on the dynamic conditional data to obtain a dynamic conditional noise tensor; inputting the static conditional data and the dynamic conditional noise tensor into a multi-conditional controlled diffusion model for conditional diffusion reconstruction processing, outputting several reconstructed video segments, and generating a reconstructed video based on the several reconstructed video segments. This application's embodiments leverage the high-fidelity generation capabilities of a multi-condition controlled diffusion model. At low bitrates, it avoids issues like blockiness and blurring common in traditional methods, generating video content more consistent with human visual perception. By combining static and dynamic condition control, it explicitly models spatiotemporal semantics during compression, enabling fine-tuning of perceptual quality and compression ratio, enhancing detail reproduction, and thus improving both perceptual quality and compression efficiency. Simultaneously, by embedding static and dynamic condition data into the diffusion process, static conditions enhance the visual accuracy and semantic consistency of static scenes, while dynamic conditions enhance the temporal consistency of dynamic scenes. These two factors work together to ensure visual and semantic accuracy of reconstructed frames, while maintaining inter-frame coherence, avoiding flickering or motion breaks, and accurately restoring the semantic and motion features of the video. Furthermore, the multi-condition controlled diffusion model supports flexible configuration of dynamically adjusted compression ratios and perceptual weights, thereby improving adaptability to different application scenarios.

[0141] To explain in detail the principles of the technical solution of this application, the overall process of this application will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principles of this application and should not be regarded as a limitation of this application.

[0142] Please see Figure 8 , Figure 8 This is a flowchart illustrating a video processing method based on a multi-condition controlled diffusion model provided in an embodiment of this application, as shown below. Figure 8 As shown, the overall implementation process of a video processing method based on a multi-condition controlled diffusion model provided in this application embodiment is mainly divided into three stages: First, keyframe selection and video segment segmentation, i.e., structuring the original video; second, multi-condition feature extraction and compression, i.e., converting various key features into a compact form; and finally, video frame reconstruction is completed at the decoding end using the multi-condition controlled diffusion model. The specific implementation process is as follows: steps 801 to 803:

[0143] Step 801, Keyframe Selection and Video Segment Segmentation:

[0144] (1) First, keyframes are selected to obtain a keyframe sequence. The selection of keyframes adopts a dual-criteria detection strategy to balance compression efficiency and reconstruction quality, specifically including the following two parts:

[0145] 1) Lens boundary detection: The TransNetV2 model is used to calculate the probability of inter-frame lens switching. ,when A frame with a value greater than 0.5 is marked as a keyframe.

[0146] 2) Fixed interval sampling: Keyframes are forcibly inserted every w frames (w is an adjustable hyperparameter) to balance compression ratio and reconstruction quality.

[0147] (2) Then, the original video is segmented based on keyframes. The segmentation algorithm is implemented based on the keyframe sequence obtained above, specifically: keyframe sequence... The original video is divided into M independent segments. It is worth mentioning that if a keyframe is triggered by a camera cut, the starting frame of its adjacent segment is shifted one frame backward to avoid ambiguity in the transition frame.

[0148] The core function of this keyframe selection and video segmentation stage is to ensure the continuity of motion within each segment, providing a spatiotemporal alignment framework for subsequent multi-condition feature extraction and diffusion reconstruction.

[0149] Step 802, Multi-condition feature extraction and compression:

[0150] To achieve precise video reconstruction control, the system extracts features from both static and dynamic dimensions and performs compression processing.

[0151] (1) Static conditions include two parts: keyframe compression and semantic description.

[0152] 1) Keyframe compression: Use image encoders such as LIC to compress the first and last keyframes while preserving visual anchor points.

[0153] 2) Semantic description (i.e., text description): Text description is generated through a multimodal model. It should be noted that the multimodal model can adopt relevant mature models in this technical field to generate text descriptions in videos, and this application embodiment does not limit this.

[0154] (2) The dynamic conditions include three parts: segmentation sequence, human motion, and optical flow field:

[0155] 1) Segmentation sequence: First, Mask2Former is used to extract the panoramic segmentation image of each frame. Then, the object contour is extracted from each panoramic segmentation image. Next, an 8th-order Bézier curve is used to fit each contour. During transmission, only the control points of the Bézier curve are transmitted. The number of curves, N, greatly improves the compression ratio.

[0156] 2) Human motion: First, SMPL is used to extract 3D joints. Then, the 3D joints are projected into 2D coordinates and small-scale movements are filtered (based on a preset movement area threshold ξ). Finally, the coordinates of 21 key joints are retained.

[0157] 3) Optical flow field: Dense optical flow is calculated based on RAFT, according to the step size. Sparse sampling, transmission arrow-like representation (direction = motion angle, length = displacement).

[0158] Among them, the dynamic conditions and their compressed representations at different thresholds are as follows: Figure 5 As shown.

[0159] Step 803, Conditional Diffusion Reconstruction at the Decoding End:

[0160] like Figure 7 As shown, the core task of the decoding end is to reconstruct video segments based on the compression conditions of transmission, and its technical implementation is based on the controllable diffusion model.

[0161] (1) Conversion of dynamic conditions to visual modalities and latent variable encoding: First, dynamic conditions such as optical flow field, segmentation sequence, and human motion are converted into visual modalities. Specifically: the optical flow field maps motion vectors to an RGB image sequence through color encoding; the segmentation sequence renders the contours fitted by Bézier curves into a binary mask sequence; and human motion forms a skeleton sequence by connecting 2D joints and is visualized as a motion trajectory graph that evolves over time. Subsequently, these visual modal data—optical flow field, segmentation sequence, and human motion—are input into a pre-trained 3D-VAE encoder. Visual modal data, such as optical flow field, segmentation sequence, and human motion, are encoded as initial low-dimensional latent variables. , {Optical flow field, segmentation sequence, human motion}.

[0162] (2) In order to achieve balanced utilization of multiple conditions, a dynamic condition control strategy is adopted to first determine the target latent variables. and target latent variables The noise is concatenated and then input into the diffusion model. Specifically, on the one hand, a random discarding mechanism is applied to each condition: a certain condition is randomly masked with a probability of 0.3 and replaced with a zero tensor, whose visual modality representation is a sequence of pure black images, to prevent overfitting. On the other hand, a modality embedding vector is introduced. Distinguishing role states based on conditions: Adding modal embedding vectors Distinguishing normal conditions ( ) and discarded state ( ):

[0163] ;

[0164] Specifically, first, the modality embedding vector With initial low-dimensional latent variables Adding them together yields the target low-dimensional latent variable that can be used as the final input. : = + , Optical flow, segmentation sequence, and human motion; then, the three target low-dimensional latent variables corresponding to these three modalities are... , {Optical flow, segmentation sequence, human motion} and noise are concatenated along the feature channel dimension to obtain a dynamic conditional noise tensor, which is then used as the model input.

[0165] (3) The diffusion process is based on the pre-trained FL2V (first frame & last frame to video) diffusion model WAN2.1, whose original inputs include the first frame, the last frame, and text descriptions. Its framework is implemented by: encoding the three target low-dimensional latent variables obtained through dynamic conditional encoding. The dynamic conditional noise tensor, obtained by concatenating optical flow, segmentation sequence, human motion, and noise along the feature channel dimension, serves as the model input. Initial conditions are the first and last keyframes of the segment, along with text descriptions. An iterative denoising process is performed through a noise prediction network to obtain denoised output features. These features are then remapped to the video RGB space by a video RGB decoding network, progressively generating intermediate frames to ultimately reconstruct the entire segment. This process, by fusing static keyframes and dynamic conditional signals, accurately restores the semantic and motion features of the video while maintaining spatiotemporal coherence.

[0166] It should be noted that this embodiment is only a brief illustrative description of the overall process of a video processing method based on a multi-condition controlled diffusion model. Detailed descriptions of each step can be found in the relevant content of the foregoing embodiments, and will not be repeated here. It is understood that this application does not impose any limitations on this.

[0167] This application embodiment acquires the original video and performs frame extraction processing on the original video using a dual-criteria detection strategy to obtain a target keyframe sequence; it then uses a segmentation algorithm to segment the original video according to the target keyframe sequence to obtain several original video segments; it performs static feature extraction and compression on each original video segment to obtain static conditional data, which includes keyframe compressed data and text description data; it then performs dynamic feature extraction and compression on each original video segment to obtain dynamic conditional data, which includes segmentation sequences, human motion data, and optical flow field data; it performs data preprocessing on the dynamic conditional data to obtain a dynamic conditional noise tensor; and it inputs the static conditional data and the dynamic conditional noise tensor into a multi-conditional controlled diffusion model for conditional diffusion reconstruction processing, outputting several reconstructed video segments, and generating a reconstructed video based on these reconstructed video segments. This application's embodiments leverage the high-fidelity generation capabilities of a multi-condition controlled diffusion model. At low bitrates, it avoids issues like blockiness and blurring common in traditional methods, generating video content more consistent with human visual perception. By combining static and dynamic condition control, it explicitly models spatiotemporal semantics during compression, enabling fine-tuning of perceptual quality and compression ratio, enhancing detail reproduction, and thus improving both perceptual quality and compression efficiency. Simultaneously, by embedding static and dynamic condition data into the diffusion process, static conditions enhance the visual accuracy and semantic consistency of static scenes, while dynamic conditions enhance the temporal consistency of dynamic scenes. These two factors work together to ensure visual and semantic accuracy of reconstructed frames, while maintaining inter-frame coherence, avoiding flickering or motion breaks, and accurately restoring the semantic and motion features of the video. Furthermore, the multi-condition controlled diffusion model supports flexible configuration of dynamically adjusted compression ratios and perceptual weights, thereby improving adaptability to different application scenarios.

[0168] To verify the effectiveness of the video compression method based on a multi-condition controlled diffusion model proposed in this application, the embodiments of this application are designed with different compression requirements (high compression ratio, medium compression ratio, and low compression ratio), and the following three embodiments are set up, each corresponding to different parameter configurations and application scenarios. The specific details are as follows:

[0169] Example 1, high compression level (suitable for scenarios with extremely limited bandwidth), the visual modal visualization of its three compression representations is as follows: Figure 5 The key parameters for Level 1 are configured as follows:

[0170] A. Keyframes are sampled at a fixed interval, w=81;

[0171] B. Number of Bézier curves N=10: 8th order Bézier curves are fitted to the contours of each frame of panoramic segmentation image, and only the control parameters corresponding to the 10 longest contours are retained to maximize compression efficiency while ensuring the topology of the core object.

[0172] C. Optical flow downsampling ratio l = 128: A large step sampling strategy is adopted, and the transmission is represented by arrows (direction represents the motion angle, and length represents the displacement), which greatly reduces the amount of optical flow data;

[0173] D. Human pose proportion threshold: Set the height of the bounding rectangle of the human pose to be greater than 1 / 5 of the video height, retain only the human poses that account for a large proportion in the image, filter out small-scale or distant human figures, and reduce redundant motion data.

[0174] Example 2, medium compression level (suitable for mainstream network environments), and the visual modal visualization of its corresponding three compression representations are as follows: Figure 5 The key parameters for Level 2 are configured as follows:

[0175] A. Keyframes are sampled at a fixed interval, w=81;

[0176] B. Number of Bézier curves N=20: The longest 20 contours in each frame are retained, achieving a balance between compression efficiency and contour integrity, and more accurately restoring the edge structure of objects;

[0177] C. Optical flow downsampling ratio = 96: It can capture the main motion trend without excessively increasing the amount of data transmission;

[0178] D. Human posture proportion threshold: The height of the bounding rectangle of the human posture is greater than 1 / 8 of the video height, so as to retain more medium-scale human postures and ensure the continuity of daily activities (such as walking and talking).

[0179] Example 3, low compression level (suitable for high-quality video transmission), the visual modal visualization of its three compression representations is as follows: Figure 5 The key parameters for Level 3 are configured as follows:

[0180] A. Keyframes are sampled at a fixed interval w=45;

[0181] B. Number of Bézier curves N=30: Retain the longest 30 contours in each frame to finely restore the edge details of objects (such as texture boundaries and small structures) and improve the sharpness of the image;

[0182] C. Optical flow downsampling ratio = 64: Optical flow vectors are sampled at 64×64 pixel block intervals to capture subtle movements (such as changes in facial expressions and slight displacement of objects) and enhance motion smoothness;

[0183] D. Human posture proportion threshold: The height of the bounding rectangle of the human posture is greater than 1 / 10 of the video height, so as to retain more small-scale human postures and ensure the accurate reproduction of complex movements (such as dance and sports competitions).

[0184] To compare performance, for each implementation (Example 1, Example 2, and Example 3), the specific performance was evaluated using two metrics, FVD and LPIPS, on the self-collected dataset, as shown in Table 1 below. Level 0 represents the baseline method, which is the method without multi-condition control (without optical flow, segmentation sequence, or human motion).

[0185] Table 1

[0186]

[0187] As shown in Table 1, the method proposed in this application (“Ours” in Table 1) can achieve better spatiotemporal coherence and visual perception quality at the same bit rate in video compression-reconstruction tasks, and has significant performance advantages compared with the current mainstream compression-reconstruction methods.

[0188] In summary, in this application embodiment, addressing the core issues of current video compression technologies such as insufficient perceptual quality at low bitrates, poor consistency in dynamic scene reconstruction, and difficulty in coordinating bitrate-distortion-perceptual optimization, this application proposes a video processing method based on a multi-condition control diffusion model, aiming to achieve the following objectives:

[0189] (1) Resolving the contradiction between perceived quality and compression efficiency: By leveraging the high-fidelity generation capability of the diffusion model, the block artifacts and blurring issues of traditional methods are avoided at low bitrates, generating video content that is more in line with human visual perception. At the same time, by combining static (keyframes, semantic description) and dynamic (optical flow, human pose, segmentation) condition control, spatiotemporal semantics are explicitly modeled during the compression process, improving the ability to restore details.

[0190] (2) Enhance the temporal consistency of dynamic scenes: Design a layered conditional fusion mechanism to embed long-term global context (such as scene semantics) and short-term local motion (such as optical flow) into the diffusion process to ensure inter-frame coherence and avoid flickering or motion breakage.

[0191] (3) Achieve efficient and controllable compression and reconstruction: Propose lightweight conditional characterization methods (such as Bézier curve fitting for contour segmentation and sparse optical flow sampling) to reduce transmission overhead and balance computational efficiency and reconstruction quality. At the same time, the model supports flexible configuration of dynamic adjustment of compression ratio and perception weight, which can adapt to different application scenarios (such as real-time communication and high-fidelity storage).

[0192] Based on the above three technical aspects, this application aims to achieve perception-priority reconstruction under high compression ratio by using a fusion diffusion model and a multi-condition control architecture. Ultimately, it aims to achieve visual quality superior to traditional coding under high compression ratio, while addressing the shortcomings of neural generation methods in terms of efficiency and controllability, thus providing a new paradigm for next-generation video compression technology.

[0193] Therefore, the method provided in this application has the following key advantages:

[0194] (1) A video compression framework with an integrated diffusion model, which controls the diffusion process through static conditions (adaptive keyframes, semantic text) and dynamic conditions (panoramic segmentation, human motion, optical flow) to achieve fine control of perceptual quality and compression ratio.

[0195] (2) A key frame selection method based on lens boundary detection and fixed interval sampling, and the resulting video segmentation strategy, can avoid generation distortion during scene transition.

[0196] (3) A multi-condition feature compression method, which uses Bézier curve fitting for segmented sequences, 2D joint filtering for human motion, and interval sampling for optical flow to achieve efficient transmission of conditional information.

[0197] (4) A dynamic condition control strategy for the decoding end, which introduces a strategy of random loss (proportion 0.3) and modal embedding (distinguishing between lost conditions and normal conditions) in the training of the diffusion model to solve the problem of multi-condition modal entanglement and ensure that the model makes balanced use of each condition.

[0198] (5) A dynamic video compression method based on key frames and conditional features to calculate the compression bit rate, which can dynamically adjust various parameters according to the video content to adapt to different bandwidth requirements.

[0199] Please see Figure 9 This application also provides a video processing apparatus 900 based on a multi-condition control diffusion model, which can implement the above-described method. The apparatus includes the following modules:

[0200] The keyframe sequence extraction module 901 is used to acquire the original video and perform frame extraction processing on the original video using a dual-criteria detection strategy to obtain the target keyframe sequence.

[0201] The video segmentation module 902 is used to segment the original video according to the target keyframe sequence using a segmentation algorithm to obtain several original video segments.

[0202] The static condition data acquisition module 903 is used to extract and compress static features from each of the original video segments to obtain static condition data; wherein, the static condition data includes keyframe compressed data and text description data;

[0203] The dynamic condition data acquisition module 904 is used to extract and compress dynamic features from each of the original video segments to obtain dynamic condition data; wherein, the dynamic condition data includes segmentation sequences, human motion data and optical flow field data;

[0204] The conditional noise tensor acquisition module 905 is used to preprocess the dynamic conditional data to obtain the dynamic conditional noise tensor.

[0205] The conditional diffusion reconstruction module 906 is used to input the static conditional data and the dynamic conditional noise tensor into a multi-conditional controlled diffusion model for conditional diffusion reconstruction processing, output several reconstructed video segments, and generate a reconstructed video based on the several reconstructed video segments.

[0206] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0207] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0208] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0209] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0210] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0211] The memory 1002 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 using the methods described in the embodiments of this application.

[0212] Input / output interface 1003 is used to implement information input and output;

[0213] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0214] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0215] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0216] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0217] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0218] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0219] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0220] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0221] This application provides a video processing, apparatus, electronic device, storage medium, and program product based on a multi-condition controlled diffusion model. It acquires an original video and performs frame extraction processing on the original video using a dual-criteria detection strategy to obtain a target keyframe sequence. A segmentation algorithm is then used to segment the original video according to the target keyframe sequence, resulting in several original video segments. Static feature extraction and compression are performed on each original video segment to obtain static conditional data, which includes keyframe compression data and text description data. Dynamic feature extraction and compression are then performed on each original video segment to obtain dynamic conditional data, which includes segmentation sequences, human motion data, and optical flow field data. The dynamic conditional data is preprocessed to obtain a dynamic conditional noise tensor. The static conditional data and the dynamic conditional noise tensor are input into a multi-condition controlled diffusion model for conditional diffusion reconstruction processing, outputting several reconstructed video segments, and a reconstructed video is generated based on these reconstructed video segments. This application's embodiments leverage the high-fidelity generation capabilities of a multi-condition controlled diffusion model. At low bitrates, it avoids issues like blockiness and blurring common in traditional methods, generating video content more consistent with human visual perception. By combining static and dynamic condition control, it explicitly models spatiotemporal semantics during compression, enabling fine-tuning of perceptual quality and compression ratio, enhancing detail reproduction, and thus improving both perceptual quality and compression efficiency. Simultaneously, by embedding static and dynamic condition data into the diffusion process, static conditions enhance the visual accuracy and semantic consistency of static scenes, while dynamic conditions enhance the temporal consistency of dynamic scenes. These two factors work together to ensure visual and semantic accuracy of reconstructed frames, while maintaining inter-frame coherence, avoiding flickering or motion breaks, and accurately restoring the semantic and motion features of the video. Furthermore, the multi-condition controlled diffusion model supports flexible configuration of dynamically adjusted compression ratios and perceptual weights, thereby improving adaptability to different application scenarios.

[0222] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0223] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0224] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0225] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0226] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0227] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0228] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0229] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0230] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0231] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0232] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A video processing method based on a multi-condition controlled diffusion model, characterized in that, The method includes the following steps: The original video is acquired, and a dual-criteria detection strategy is used to extract frames from the original video to obtain the target keyframe sequence. A segmentation algorithm is used to segment the original video based on the target keyframe sequence to obtain several original video segments; Static feature extraction and compression are performed on each of the original video segments to obtain static conditional data; wherein, the static conditional data includes keyframe compressed data and text description data; Dynamic features are extracted and compressed from each of the original video segments to obtain dynamic conditional data; wherein, the dynamic conditional data includes segmentation sequences, human motion data, and optical flow field data; The dynamic condition data is preprocessed to obtain the dynamic condition noise tensor; The static condition data and the dynamic condition noise tensor are input into a multi-condition controlled diffusion model for conditional diffusion reconstruction processing, outputting several reconstructed video segments, and a reconstructed video is generated based on the several reconstructed video segments. The step of preprocessing the dynamic condition data to obtain the dynamic condition noise tensor includes: The dynamic conditional data is converted into visual modal data, and the visual modal data is input into a pre-trained encoder for encoding processing to obtain initial low-dimensional latent variables; A dynamic condition control strategy is adopted to transform the initial low-dimensional latent variable into a target low-dimensional latent variable, and then concatenate the target low-dimensional latent variable with noise to obtain the dynamic condition noise tensor. The dynamic condition control strategy includes a random drop mechanism and a modal embedding mechanism. The modal embedding mechanism is used to distinguish the role states of the condition, and the role states include normal conditions and dropout states.

2. The method according to claim 1, characterized in that, The dual-criteria detection strategy includes a shot boundary detection method and a fixed-interval sampling method. The process of acquiring the original video and using the dual-criteria detection strategy to extract frames from the original video to obtain a target keyframe sequence includes: Obtain the original video; Using the aforementioned shot boundary detection method, the inter-frame shot switching probability in the original video is calculated, and the first keyframe sequence is determined based on the inter-frame shot switching probability and a preset switching probability threshold. Using the fixed-interval sampling method, keyframes are marked in the original video according to a preset frame interval to obtain a second keyframe sequence; The target keyframe sequence is constructed based on the first keyframe sequence and the second keyframe sequence.

3. The method according to claim 1, characterized in that, The step of extracting and compressing static features from each of the original video segments to obtain static conditional data includes: Frame feature extraction is performed on each of the original video segments to obtain the first keyframe and the last keyframe of each of the original video segments. The first and last keyframes in each of the original video segments are compressed using an image encoder to obtain the keyframe compressed data. The original video segments are processed by a multimodal model to extract text, thereby obtaining the text description data. The static condition data is constructed based on the keyframe compressed data and the text description data.

4. The method according to claim 1, characterized in that, The process of extracting and compressing dynamic features from each of the original video segments to obtain dynamic conditional data includes: Panoramic segmentation is performed on the video frames in each of the original video segments to obtain a panoramic segmentation map of each video frame, and the object contours of the panoramic segmentation map of each video frame are extracted. The segmentation sequence is obtained by fitting the contour of each object using a Bézier curve; Extract the three-dimensional joint sequence from each of the original video segments, and project the three-dimensional joint sequence from each of the original video segments into two-dimensional coordinate data; Based on a preset motion area threshold, the two-dimensional coordinate data in each of the original video segments are filtered to obtain the human motion data; Calculate the dense optical flow in each of the original video segments to obtain the optical flow field data; The dynamic condition data is constructed based on the segmentation sequence, the human motion data, and the optical flow field data.

5. The method according to claim 1, characterized in that, The process of converting the dynamic conditional data into visual modal data and inputting the visual modal data into a pre-trained encoder for encoding processing to obtain initial low-dimensional latent variables includes: The object contours in the segmented sequence after Bézier curve fitting are rendered as a binary mask sequence; Connect the two-dimensional joints in the human motion data to generate a skeleton sequence, and visualize the skeleton sequence as a motion trajectory diagram that evolves over time. The motion vectors in the optical flow field data are mapped to an RGB image sequence using a color encoding method; The visual modality data is constructed based on the binary mask sequence, the motion trajectory map, and the RGB image sequence. The visual modality data is input into the pre-trained encoder for encoding processing to obtain the initial low-dimensional latent variables.

6. A video processing device based on a multi-condition control diffusion model, characterized in that, The device includes the following modules: The keyframe sequence extraction module is used to acquire the original video and perform frame extraction processing on the original video using a dual-criteria detection strategy to obtain the target keyframe sequence. The video segmentation module is used to segment the original video according to the target keyframe sequence using a segmentation algorithm to obtain several original video segments; The static condition data acquisition module is used to extract and compress static features from each of the original video segments to obtain static condition data; wherein, the static condition data includes keyframe compressed data and text description data; The dynamic condition data acquisition module is used to extract and compress dynamic features from each of the original video segments to obtain dynamic condition data; wherein, the dynamic condition data includes segmentation sequences, human motion data, and optical flow field data; The conditional noise tensor acquisition module is used to preprocess the dynamic conditional data to obtain the dynamic conditional noise tensor. The conditional diffusion reconstruction module is used to input the static conditional data and the dynamic conditional noise tensor into a multi-conditional controlled diffusion model for conditional diffusion reconstruction processing, output several reconstructed video segments, and generate a reconstructed video based on the several reconstructed video segments. Specifically, the conditional noise tensor acquisition module is used for: The dynamic conditional data is converted into visual modal data, and the visual modal data is input into a pre-trained encoder for encoding processing to obtain initial low-dimensional latent variables; A dynamic condition control strategy is adopted to transform the initial low-dimensional latent variable into a target low-dimensional latent variable, and then concatenate the target low-dimensional latent variable with noise to obtain the dynamic condition noise tensor. The dynamic condition control strategy includes a random drop mechanism and a modal embedding mechanism. The modal embedding mechanism is used to distinguish the role states of the condition, and the role states include normal conditions and dropout states.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images