Video processing method, apparatus, device, storage medium and program product

Abstract

The present disclosure provides a video processing method, device, equipment, storage medium and program product, relating to the technical field of computer. The method comprises: acquiring a source video and a multi-modal control condition corresponding to the source video, wherein the multi-modal control condition comprises depth information representing a scene geometric structure, contour information representing a target contour, and a segmentation mask of a video frame; performing feature encoding on the source video to obtain a denoising latent feature; performing feature encoding on the multi-modal control condition to obtain a control latent feature; guiding the denoising latent feature using the control latent feature to predict and remove noise in the denoising latent feature to obtain a target latent feature; and performing decoding processing on the target latent feature to obtain a target video, wherein the visual quality of the target video is stronger than that of the source video. The method can improve the visual enhancement effect of the video.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a video processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] Visual enhancement technology is an important research direction in the fields of digital image processing and computer vision. Its core goal is to restore or enhance the visual experience of low-quality or degraded visual content by processing it, thereby meeting the needs of human observation or machine analysis.

[0003] In real-world applications, video data is often affected by various factors, leading to quality degradation. Especially in open environments such as outdoor surveillance, vehicle driving detection, or mobile filming, acquisition equipment is inevitably constrained by environmental factors. Under adverse conditions such as rain, snow, fog, haze, or low light, video quality suffers severe degradation, greatly limiting the effectiveness of subsequent video analysis and applications.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a video processing method, apparatus, device, storage medium, and program product.

[0006] According to a first aspect of the present disclosure, a video processing method is provided, comprising: acquiring a source video and multimodal control conditions corresponding to the source video, wherein the multimodal control conditions include depth information representing the geometric structure of a scene, contour information representing the contour of a target, and a segmentation mask of a video frame; performing feature encoding on the source video to obtain potential features to be denoised; performing feature encoding on the multimodal control conditions to obtain control potential features; using the control potential features to guide the potential features to be denoised in order to predict and remove noise in the potential features to be denoised, thereby obtaining target potential features; and performing decoding processing on the target potential features to obtain a target video, wherein the visual quality of the target video is stronger than that of the source video.

[0007] In some implementations, the control latent features are used to guide the latent features to be denoised in order to predict and remove noise from the latent features to be denoised, thereby obtaining the target latent features. This includes: based on the current latent features to be denoised and the control latent features, performing noise prediction and removing the predicted noise to obtain updated latent features to be denoised; based on the updated latent features to be denoised and the control latent features, performing noise prediction and removing the predicted noise, until a preset number of iterations are completed or a preset stopping condition is met, thereby obtaining the target latent features.

[0008] In some implementations, based on the current potential features to be denoised and the control potential features, noise prediction is performed and the predicted noise is removed to obtain updated potential features to be denoised. This includes: concatenating or adding the potential features to be denoised and the control potential features to obtain fused features; inputting the fused features into a denoising network, which, guided by the control potential features, predicts the noise in the potential features to be denoised, removes the noise from the potential features to be denoised, and outputs updated potential features to be denoised.

[0009] In some implementations, the denoising network employs a diffusion model network that includes rotational position coding, which is used to encode the positional information of the fused features in the spatial and temporal dimensions.

[0010] In some implementations, the multimodal control conditions also include radar information; the method further includes: acquiring text prompts as semantic guidance input; embedding and encoding the text prompts to obtain text latent features; and performing noise prediction and removing predicted noise based on the current latent features to be denoised and the control latent features, including: using the control latent features and text latent features to jointly predict noise for the current latent features to be denoised and removing the predicted noise.

[0011] In some implementations, the method further includes: obtaining a guiding scale and an inference step count, wherein the guiding scale is used to control the degree to which the target video conforms to multimodal control conditions, and the inference step count is used to control the number of denoising iterations.

[0012] In some implementations, feature encoding is performed on the multimodal control conditions to obtain control latent features, including: performing independent feature encoding on depth information, contour information, and segmentation masks of video frames to obtain corresponding depth latent features, contour latent features, and segmentation latent features; and performing feature fusion on the depth latent features, contour latent features, and segmentation latent features to obtain control latent features.

[0013] According to a second aspect of the present disclosure, a video processing apparatus is provided, including an acquisition module, a first encoding module, a second encoding module, a noise reduction guidance module, and a decoding module.

[0014] The acquisition module is used to acquire the source video and the corresponding multimodal control conditions. The multimodal control conditions include depth information representing the geometric structure of the scene, contour information representing the contour of the target, and segmentation mask of the video frame. The first encoding module is used to perform feature encoding on the source video to obtain the potential features to be denoised; The second encoding module is used to encode the multimodal control conditions to obtain the latent control features; The denoising guidance module is used to guide the latent features to be denoised by controlling the latent features, so as to predict and remove noise in the latent features to be denoised, and obtain the target latent features. The decoding module is used to decode the latent features of the target to obtain the target video, which has a higher visual quality than the source video.

[0015] In some implementations, the denoising guidance module includes an iterative denoising unit and an iterative control unit.

[0016] The iterative denoising unit is used to predict noise and remove the predicted noise based on the current latent features to be denoised and the control latent features, so as to obtain the updated latent features to be denoised. The iterative control unit is used to predict noise and remove predicted noise based on the updated latent features to be denoised and control latent features, until a preset number of iterations are completed or a preset stopping condition is met, to obtain the target latent features.

[0017] In some implementations, the iterative denoising unit is used to concatenate or add the latent features to be denoised with the control latent features to obtain fused features; the fused features are input into the denoising network, which predicts the noise in the latent features to be denoised based on the guidance of the control latent features, removes the noise from the latent features to be denoised, and outputs the updated latent features to be denoised.

[0018] In some implementations, the denoising network employs a diffusion model network that includes rotational position coding, which is used to encode the positional information of the fused features in the spatial and temporal dimensions.

[0019] In some implementations, the multimodal control conditions also include radar information; the device also includes a text acquisition module and a text encoding module.

[0020] The text acquisition module is used to acquire text prompts as semantic guidance input; The text encoding module is used to embed and encode text prompts to obtain latent text features; The iterative denoising unit is used to predict noise for the current latent features to be denoised by using control latent features and text latent features together, and remove the predicted noise.

[0021] In some implementations, the acquisition module further includes a parameter acquisition unit.

[0022] The parameter acquisition unit is used to acquire the guiding scale and the number of inference steps. The guiding scale is used to control the degree to which the target video follows the multimodal control conditions, and the number of inference steps is used to control the number of denoising iterations.

[0023] In some implementations, the second encoding module is used to independently encode the depth information, contour information, and segmentation mask of the video frame to obtain the corresponding depth latent features, contour latent features, and segmentation latent features; and to perform feature fusion on the depth latent features, contour latent features, and segmentation latent features to obtain control latent features.

[0024] According to a third aspect of the present disclosure, a vehicle is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the video processing method described above.

[0025] According to a fourth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the video processing method described above.

[0026] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, which, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform the video processing method described above.

[0027] According to a sixth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the video processing method described above.

[0028] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: This disclosure acquires the source video and its corresponding multimodal control conditions, and uses depth information, contour information, and segmentation masks of video frames to represent the geometric structure, target contour, and semantic region of the scene, respectively, thus constructing a three-dimensional, multi-dimensional constraint on the video content. In the feature processing stage, feature encoding is performed on the source video and multimodal control conditions respectively, realizing the decoupled representation of content features and control features, laying the foundation for subsequent accurate guidance. In the denoising process, the control latent features are used to guide the latent features to be denoised, which essentially integrates prior knowledge such as the geometric structure, target contour, and semantic region of the scene into the noise prediction and removal process. This makes the denoising operation no longer rely solely on pixel-level statistical characteristics, but can be optimized in a targeted manner based on the structured information of the scene. This multimodal guidance mechanism can effectively constrain the generation process and avoid the common problems of detail blurring, structural distortion, or semantic inconsistency in traditional denoising methods, ensuring that while removing noise, the geometric shape of the scene is accurately restored, the contour details of the target are preserved, and the coherence of the semantic region is maintained. The final decoded target video not only outperforms the source video in terms of signal-to-noise ratio, but also achieves significant improvements in structural integrity, detail clarity, and semantic rationality.

[0029] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0031] Figure 1 This is a flowchart illustrating a video processing method according to some embodiments of the present disclosure.

[0032] Figure 2 This is a flowchart illustrating the determination of potential control features according to some embodiments of the present disclosure.

[0033] Figure 3 This is a flowchart illustrating the determination of potential features of a target according to some embodiments of the present disclosure.

[0034] Figure 4 This is a flowchart illustrating a feature denoising method according to some embodiments of the present disclosure.

[0035] Figure 5 This is a flowchart illustrating another video processing method according to some embodiments of the present disclosure.

[0036] Figure 6 This is a schematic diagram illustrating the training process of a video processing model according to some embodiments of the present disclosure.

[0037] Figure 7 This is a schematic diagram illustrating the structure of a DiT module and its included cross-attention module according to some embodiments of the present disclosure.

[0038] Figure 8 This is a schematic diagram illustrating the use of a video processing model according to some embodiments of the present disclosure.

[0039] Figure 9 This is a block diagram illustrating a video processing apparatus according to some embodiments of the present disclosure.

[0040] Figure 10 This is a block diagram illustrating an electronic device according to some embodiments of the present disclosure. Detailed Implementation

[0041] Exemplary embodiments of this disclosure will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0042] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all content and steps, nor does it necessarily have to be executed in the described order or in the order of the step numbers. For example, some steps can be broken down, while others can be combined or partially combined, and multiple steps can have their order interchanged or be executed simultaneously. Therefore, the actual execution order may change depending on the actual situation.

[0043] The embodiments described below, which are examples of some of the embodiments of this disclosure, do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0044] In real-world applications, video data is often affected by various factors, leading to quality degradation. Especially in open environments such as outdoor surveillance, vehicle driving detection, or mobile filming, acquisition equipment is inevitably constrained by environmental factors. Under adverse conditions such as rain, snow, fog, haze, or low light, video quality suffers severe degradation, greatly limiting the effectiveness of subsequent video analysis and applications.

[0045] As described in the background section, in open environments such as outdoor monitoring, vehicle driving perception, or mobile shooting, the acquisition equipment is inevitably constrained by environmental factors, and the video quality will suffer severe degradation, which greatly limits the subsequent video analysis and application effects.

[0046] In some embodiments, under adverse conditions such as rain, snow, fog, haze, or low light, the vehicle's video acquisition equipment is often severely interfered with, and the video quality is severely degraded, resulting in a decrease in the recognition accuracy of traffic elements (such as vehicles, pedestrians, or road signs), which seriously affects the safety and reliability of the assisted driving system.

[0047] In some embodiments, structure-guided reconstruction techniques can be employed to generate visually enhanced videos. These methods focus on extracting core structural information from video sequences, such as edge lines and depth maps, to guide the reconstruction from a bad weather view to a clear scene. However, when dealing with complex dynamic scenes, these methods still require optimization of the accuracy of structure extraction to reduce reliance on computational resources.

[0048] In some embodiments, a multi-source data-driven approach can also be used to generate visually enhanced videos. This approach emphasizes fusing multiple sensor inputs (such as LiDAR point clouds, depth maps, and camera video) to assist in the synthesis of clear-sight videos, overcoming the limitations of single-modal approaches in low visibility conditions.

[0049] The aforementioned visual enhancement solutions generally suffer from problems such as insufficient geometric and semantic consistency across multiple perspectives, instability in handling long-term dynamic objects, limited support for complex adverse weather conditions and controllable enhancements, and a high dependence on high-precision sensors during deployment.

[0050] In some embodiments of this disclosure, the data involved includes, but is not limited to, the data itself, its acquisition, use, updating, analysis, processing, transmission, and storage, and shall comply with the laws and regulations of the country where the data is located.

[0051] In some embodiments of this disclosure, data, information, etc., may be obtained after obtaining the user's consent.

[0052] The specific implementation methods of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0053] Figure 1 This is a flowchart illustrating a video processing method according to some embodiments of the present disclosure. This video processing method can be applied to electronic devices, including but not limited to in-vehicle terminals, smartphones, smart tablets, wearable devices, desktop computers, laptops, and other terminal devices. It can also include server-side components such as local servers and cloud servers, which can be deployed in a computer cluster or a combination of multiple computers. Figure 1 As shown, the video processing method may include S101-S105.

[0054] In S101, the source video and the corresponding multimodal control conditions are obtained. The multimodal control conditions include depth information representing the geometric structure of the scene, contour information representing the contour of the target, and segmentation mask of the video frame.

[0055] The source video refers to the input video data to be processed. In this solution, the source video can be video taken by a vehicle at night or in inclement weather (such as rain, snow, fog), which has poor visual quality (blurry, low light, noise).

[0056] Multimodal control conditions refer to auxiliary information used to constrain and guide video generation. Depth information represents the geometric structure of the scene, reflecting the distance of objects from the camera. Contour information represents the contours of targets, typically including edges and lines. The segmentation mask of a video frame is data representing semantic regions, classifying the image into different categories, such as roads, vehicles, and pedestrians. As an example, depth information can be a depth map corresponding to a video frame in the source video, and contour information can be a line drawing, also known as a line art, corresponding to a video frame in the source video.

[0057] In some embodiments, a depth estimation model can be used to process the source video frames to generate a depth map, capturing the 3D structure of the scene.

[0058] In some embodiments, edge detection algorithms can be used to extract edge lines in a video; or line detection models can be used to extract standard line graphs to preserve details such as lane lines and vehicle outlines.

[0059] In some embodiments, a semantic segmentation model or object detection model can be used to extract a mask to distinguish foreground objects (vehicles, pedestrians) from background areas, and even specific traffic signs.

[0060] In some embodiments, the video frame size can be adjusted according to the target resolution, and the segmentation mask can be remapped and dilated to enhance the weight of key areas such as road markings.

[0061] In S102, feature encoding is performed on the source video to obtain the potential features to be denoised.

[0062] The latent features to be denoised can be the representation of the source video in the latent space, and subsequent steps will denoise them in this space.

[0063] In some embodiments, the source video (an RGB video frame sequence) can be input into the encoder of a variational autoencoder (VAE). The VAE compresses high-dimensional pixel data into a low-dimensional latent space. The output of the VAE is a compact tensor, which is the latent feature to be denoised. This latent feature retains the core content information of the source video but is in a state of "noise interference," awaiting correction in subsequent steps.

[0064] In S103, feature encoding is performed on the multimodal control conditions to obtain the control latent features.

[0065] The control latent features can be the mathematical representation of the multimodal control conditions in the latent space, used to guide the generation direction of the denoising network. In some embodiments, the feature encoding of the multimodal control conditions described above can be mapping depth information, contour information, and segmentation masks of video frames into the same latent space.

[0066] In S104, the latent features to be denoised are guided by the control latent features to predict and remove noise from the latent features to be denoised, thereby obtaining the target latent features.

[0067] In some embodiments, control latent features can be input into the denoising network, and a cross-attention module can be used to make the latent features to be denoised pay attention to the control features. For example, the model can correct the corresponding lines in the latent features based on the lane line positions in the control features to ensure that the lane lines in the generated video are not curved.

[0068] In S105, the latent features of the target are decoded to obtain the target video, which has a higher visual quality than the source video.

[0069] Decoding the latent features of the target involves transforming the mathematical representation of the latent space back into the visualized pixel space. The target video is the final generated video, with visual quality superior to the source video.

[0070] The aforementioned visual quality can be considered a comprehensive evaluation metric. Improved visual quality refers not only to conventional image sharpness, but more importantly, to a significant improvement in structural integrity, spatiotemporal consistency, and detail reproduction compared to the source video. Specifically, improved visual quality means that the target video can effectively eliminate blur, noise, or artifacts caused by adverse environmental conditions (such as rain, snow, and fog) in the source video, exhibiting higher sharpness and color fidelity. Simultaneously, thanks to the guidance of multimodal control conditions, the target video strictly adheres to the geometric and semantic constraints of the scene during denoising and enhancement, ensuring the structural accuracy and distortion-free nature of key targets such as vehicles, pedestrians, and lane lines, while maintaining smooth and natural dynamic effects in the temporal dimension. This achieves comprehensive quality enhancement from perceptual perception to objective metrics.

[0071] In some embodiments, the target latent features (i.e., the clean features after denoising) can be input into the decoder of a variational autoencoder (VAE). The decoder restores the low-dimensional latent features to a high-dimensional RGB pixel space through upsampling and deconvolution operations, outputting a video with significantly enhanced visual quality. In other words, through the above steps S101-S105, a blurry and dark rainy / snowy night video (source video) can be transformed into a clear and bright "sunny daytime" video (target video) while maintaining the original scene semantics and geometric structure.

[0072] As can be seen from the above steps, when applying the above solution to the fields of intelligent driving and intelligent assisted driving, converting videos taken in adverse weather conditions (such as rain, snow, and fog) into clear, daytime footage can effectively improve the driver's visual perception, provide clearer assisted driving images, ensure driving safety, and help improve the perception and decision-making capabilities of assisted driving models in complex environments. Furthermore, for accident videos from dashcams and in sentry mode, using the solution of this disclosure to enhance the images can assist in accident playback and liability determination, improving the accuracy and efficiency of post-accident analysis.

[0073] In some embodiments of this disclosure, feature encoding of multimodal control conditions is performed to obtain latent control features, which may include... Figure 2 S201-S202 are shown.

[0074] In S201, depth information, contour information, and segmentation mask of video frames are independently feature encoded to obtain corresponding depth latent features, contour latent features, and segmentation latent features.

[0075] In some embodiments, a depth estimation network can be used to convert an RGB image into a single-channel or multi-channel depth map, and then convert it into a feature vector.

[0076] In some embodiments, Canny edge detection, HED (Holistically-Nested Edge Detection), or PidiNet algorithms can be used to extract coarse lines; or a model can be used to extract fine structural line drawings.

[0077] In some embodiments, a model can be used to generate a semantic segmentation mask that distinguishes foreground objects (such as traffic signs and pedestrians) from the background.

[0078] In some embodiments, the aforementioned depth information and contour information can be depth maps or line drawings. Although the depth maps, line drawings, and segmentation masks of the video frames mentioned earlier differ in pixel space, they need to be mapped to the same latent space. As an example, the encoder portion of a variational autoencoder (VAE) or convolutional layers in a specially designed ControlNet structure can be used to downsample and extract features from these three types of images respectively, ensuring that their output feature maps maintain consistency in size (height, width) and batch size for subsequent fusion.

[0079] In S202, feature fusion is performed on deep latent features, contour latent features, and segmentation latent features to obtain control latent features.

[0080] Feature fusion refers to merging multiple independent feature vectors into a single feature tensor so that subsequent denoising networks (such as DiT) can simultaneously "see" geometric, contour, and semantic information.

[0081] In some embodiments, feature fusion can be performed by concatenating channels.

[0082] This disclosure, through independent encoding and feature fusion, firstly avoids feature interference caused by differences in the distribution of data from different modalities, ensuring the complete extraction of depth information, contour information, and segmentation masks of video frames; secondly, the fused features integrate multi-dimensional information of scene geometry, contour, and semantics, providing precise guidance for the denoising network and significantly improving the visual quality and content consistency of the generated video; furthermore, this modular design enhances the flexibility and scalability of the solution for different control modalities.

[0083] In some embodiments, the aforementioned depth information and contour information can be depth maps and line drawings. The implementation of feature encoding of multimodal control conditions to obtain control latent features can also involve directly concatenating the depth map (1 channel), the line drawing (1 or 3 channels), and the segmentation mask of the video frame (N channels, where N is the number of classes) along the channel dimension to form a super input tensor; designing a convolutional layer (Conv Layer) or encoder specifically adapted to the number of input channels of this super tensor to map it to the latent space in one go.

[0084] In some embodiments of this disclosure, control latent features are used to guide the latent features to be denoised, predicting and removing noise from the latent features to be denoised, thereby obtaining the target latent feature. This can be achieved by predicting noise and removing predicted noise based on the current latent features to be denoised and control latent features, resulting in an updated latent feature to be denoised; then, based on the updated latent features to be denoised and control latent features, noise prediction and removal of predicted noise are performed until a preset number of iterations are completed or a preset stopping condition is met, thus obtaining the target latent feature. (See reference...) Figure 3 The process of determining the potential characteristics of the target may include S301-S303.

[0085] In S301, based on the current potential features to be denoised and the control potential features, noise is predicted and the predicted noise is removed to obtain the updated potential features to be denoised. In S302, it is determined whether the preset number of iterations has been completed or the preset stopping condition has been met; If the preset number of iterations is not completed or the preset stopping condition is not met, the process of S301-S02 continues until the preset number of iterations is completed or the preset stopping condition is met, and then enters S303 to obtain the target potential feature.

[0086] This disclosure employs an iterative denoising mechanism, utilizing controlled latent features to continuously guide noise prediction at each denoising step, ensuring the generation process strictly adheres to multimodal constraints. Gradually removing noise and updating features effectively improves denoising accuracy and video generation stability. The introduction of preset iteration counts or stopping conditions achieves a flexible balance between generation quality and computational efficiency, ultimately yielding high-fidelity, structurally consistent target latent features.

[0087] In some embodiments of this disclosure, step S301 above, based on the current potential features to be denoised and the control potential features, performs noise prediction and removes the predicted noise to obtain updated potential features to be denoised, which may include... Figure 4 S401-S402 are shown.

[0088] In S401, the latent features to be denoised and the latent control features are concatenated or added to obtain fused features; In S402, the fused features are input into the denoising network. Guided by the control latent features, the denoising network predicts the noise in the latent features to be denoised, removes the noise from the latent features to be denoised, and outputs the updated latent features to be denoised.

[0089] The aforementioned denoising network can employ a diffusion model architecture based on a 3D transformer. When the fused features are input into the network, the network's internal cross-attention mechanism comes into play, which is crucial for achieving "guidance." The network uses the latent features to be denoised as query signals and the control latent features as keys and values. By calculating the correlation between the two, the generation process closely "follows" and follows the control signals. For example, when the network processes a specific region in the image, it uses the attention mechanism to identify the corresponding "lane lines" or "buildings" information in the control features, thereby ensuring that accurate geometric structure and semantic content are preserved while denoising.

[0090] After completing the feature interaction and guidance, the network enters the noise prediction stage. After processing by multiple network modules, the model finally outputs a tensor with the same shape as the input. This tensor represents the network's prediction of the noise distribution in the current image. Understandably, the network's goal is not to directly generate a clear image, but rather to accurately calculate the noise components mixed into the image. Subsequently, using a pre-defined scheduling algorithm, based on the predicted noise data and the current time step parameters, the corresponding noise components are subtracted from the original features to be denoised. Through this subtraction operation, the originally noisy features are updated to new features with less noise and higher clarity, thus completing one step of denoising iteration and laying the foundation for the subsequent generation of high-quality video frames.

[0091] This disclosure deeply integrates control information with the features to be denoised, providing the network with explicit structural and semantic guidance. Utilizing the cross-attention mechanism in the denoising network, it achieves precise constraints of control conditions on the noise prediction process, ensuring that the denoising direction conforms to the scene's geometry and contour requirements. This method not only effectively improves the accuracy of noise prediction but also significantly enhances the spatiotemporal structural consistency and visual quality of the generated video.

[0092] In some embodiments of this disclosure, the denoising network employs a diffusion model network that includes rotational position coding, which is used to encode the positional information of the fused features in the spatial and temporal dimensions.

[0093] Rotational position encoding is a technique used to enable models to understand positional relationships within data. It treats different parts of a feature vector as complex numbers and rotates them at different angles using trigonometric functions based on their location (in video, this means specific spatial pixel and temporal frame positions).

[0094] By introducing rotational position encoding, embodiments of this disclosure enable the denoising network to accurately perceive the relative positional relationships of video features in the spatiotemporal dimension. This mechanism utilizes rotational transformation to inject positional information, which not only enhances the model's ability to model and extrapolate long video sequences, effectively avoiding performance degradation when processing high-resolution or long-duration videos, but also significantly improves the temporal coherence and spatial consistency of the generated videos.

[0095] In some embodiments of this disclosure, the multimodal control conditions further include radar information; the video processing method further includes: acquiring text prompts as semantic guidance input; embedding and encoding the text prompts to obtain text latent features. Furthermore, the above-mentioned noise prediction and removal of predicted noise based on the current latent features to be denoised and the control latent features can be achieved by using the control latent features and text latent features together to predict noise in the current latent features to be denoised and remove the predicted noise. (See reference...) Figure 5The above video processing methods may include S501-S506.

[0096] In S501, the source video and the corresponding multimodal control conditions and text prompts are obtained. The multimodal control conditions include depth information representing the geometric structure of the scene, contour information representing the contour of the target, segmentation mask of the video frame, and radar information. In S502, feature encoding is performed on the source video to obtain the potential features to be denoised; In S503, feature encoding is performed on the multimodal control conditions to obtain the control latent features; In S504, the text prompt words are embedded and encoded to obtain the latent features of the text; In S505, the control latent features and text latent features are used together to predict noise in the current latent features to be denoised, and the predicted noise is removed. In S506, the latent features of the target are decoded to obtain the target video, which has a higher visual quality than the source video.

[0097] This disclosure achieves refined guidance from geometric structure to semantic content by fusing multimodal control information from both visual and physical dimensions. The introduction of radar information compensates for the limitations of depth and velocity detection in pure visual perception. Combined with depth information, contour information, and video frame segmentation masks, it ensures the spatiotemporal structural consistency of the generated video in complex scenes. Simultaneously, the embedding and encoding of text prompts provides high-level semantic guidance. Noise prediction is performed using dual constraints of control features and text features, ensuring that the denoising process preserves accurate physical contours while enabling rich content editing and enhancement, ultimately significantly improving the visual quality and semantic alignment of the target video.

[0098] In some embodiments of this disclosure, the video processing method described above may further include obtaining a guiding scale and an inference step count, wherein the guiding scale is used to control the degree to which the target video follows multimodal control conditions, and the inference step count is used to control the number of denoising iterations.

[0099] The acquired guiding scale and inference steps are primarily used to control the behavior and depth of the denoising iteration process. These two parameters can be loaded before denoising begins. Subsequently, during the denoising loop, the guiding scale is injected into the cross-attention module of the denoising network. By adjusting the vector difference weights between conditional and unconditional predictions, it controls the strength of the target video's adherence to multimodal control conditions. Higher values ​​result in a more rigorous fit to the input conditions, but excessively high values ​​may lead to a harsh visual. The inference steps directly determine the number of loop executions, controlling the denoising precision as it evolves from pure noise to a clear video. More steps result in a more refined intermediate process and richer video details, but also increase computational costs.

[0100] By introducing a dual control mechanism of guided scale and inference steps, this disclosure achieves flexible adjustment of generation quality and control precision. The guided scale, through a weighted adjustment strategy, balances multimodal constraints with image naturalness, avoiding artifacts caused by excessive constraints; the inference steps, by controlling the denoising iteration depth, significantly improve the video's detail restoration capability and spatiotemporal consistency. The synergistic effect of these two mechanisms enables the system to generate high-fidelity, detailed target videos that strictly conform to user intent while ensuring efficient inference.

[0101] This disclosure converts self-driving videos from nighttime and inclement weather (such as rain, snow, and fog) into clear, bright video footage, effectively improving visual clarity and target recognition accuracy in the driving environment. This helps drivers better perceive road conditions and enhances driving safety. By improving video quality under adverse weather conditions and reducing the effects of insufficient light, blur, and noise, it significantly enhances the performance of the driver assistance system and visual perception module, improving the stability and reliability of the driver assistance system in complex environments. Furthermore, this disclosure's solution is also applicable to scenarios such as electronic rearview mirrors and dashcams, improving image quality at night and in adverse weather conditions, enhancing the driver's visual experience and safety.

[0102] In some embodiments, the above video processing method can be implemented using a video processing model. For ease of understanding, the training and usage process of the video processing model will be explained in detail below using a vehicle-captured video processing scenario as an example.

[0103] like Figure 6 As shown, the training inputs are the original video with added noise, a dynamic mask, a depth video, a segmented video, and a line drawing video. The output is the predicted noise. The feedback mechanism consists of the predicted noise and the mean squared error loss (MSELoss) of the added noise.

[0104] In some embodiments, Figure 6 The diffusion model can be a diffusion model based on the Transformer architecture, which includes multiple DiT (Diffusion Transformer) modules. The architecture of each DiT module is as follows: Figure 7 As shown.

[0105] The training data for this publicly disclosed video processing model can be RGB three-channel video, and the data content can be real-world multi-scene autonomous driving 7V video data.

[0106] In some embodiments, to address the problem of uneven distribution of different severe weather styles, a balanced sampling dataset can be used during training to enable a single model to support multiple weather style transitions, thereby simplifying the training and deployment process.

[0107] In some embodiments, a dynamic masking mechanism can be used to mix text-to-video (T2V) and image-to-video (IT2V) tasks during training, improving training efficiency and enabling a single model to support the generation of infinitely long videos. Based on various control conditions such as line drawings, depth maps, object detection, and segmentation maps, the video content and structure are precisely restored, enhancing the model's ability to perceive key targets and lane lines.

[0108] In some embodiments, Gaussian noise can be injected into the control conditions during the training phase to reduce the model's dependence on the original style information in the control conditions, improve the model's ability to generate the target style based on the prompt words, and achieve flexible adjustment of style intensity.

[0109] In addition, refer to Figure 7 This disclosure incorporates a style-injection cross-attention module to enhance the injection of style information from prompts during training, thereby strengthening the expression of style information during generation. A new zero-convolution module further improves video frame generation. Extended rotational position encoding supports higher resolution generation, enhancing the model's adaptability to large-sized videos while maintaining consistent generation quality. An online segmentation model replaces traditional object detection bounding boxes, supplementing key regions (such as signs and pedestrians) with information to address style anomalies and detail loss at night, further improving the quality of generated images.

[0110] like Figure 8 As shown, when processing videos using the trained model, the inputs are the original video (pure or noisy), dynamic mask, depth video, segmented video, and line drawing video. The output is the predicted noise, which is removed from the current input by the DDIM scheduler. This process is iterated n times to obtain the generated video. During the inference phase, by reweighting the style-related tokens in the self-attention module, the style intensity can be flexibly adjusted without changing the model structure and parameters. This scheme implements style reweighting for three self-attention modules: Classic, Chunk, and Flash. The Flash module significantly reduces memory usage and inference time while supporting substantial style adjustments to ensure generation quality. Furthermore, combined with a multi-view synchronous generation mechanism and the aforementioned control conditions and style control strategies, high-quality video conversion from inclement weather to sunny weather can be achieved.

[0111] The complete processing flow for a single video during the use of the above model is as follows: Initialization and Data Preparation: Load the trained model (including variational autoencoder, T5 text encoder, and 3D transformer) and control condition generator. Obtain video frames, segmentation masks (OD and LD), line plots, depth maps, and LiDAR intensity maps for specified views (e.g., center view, front view) from the input video path. Adjust the video frame size according to the target resolution and remap and dilate the segmentation masks to enhance key areas (e.g., road markings).

[0112] In some embodiments, the control condition generator can be used to perform processing steps such as edge and line drawing detection, depth estimation, object detection and segmentation, and pose estimation.

[0113] Extracting Control Conditions: A specified weak control method (e.g., Leres depth estimation) is applied to the input video frames to generate initial guiding features, capturing the scene's structural information. A strong control method (e.g., Lineart standard detection) is applied, combined with OD segmentation masks, object detection, and DeepLab segmentation (e.g., traffic signs), to generate feature maps emphasizing key objects. Background regions are either noise-added or zeroed out based on noise intensity parameters. A third control method (e.g., Pidi line detection), combined with LD segmentation masks, is applied to enhance details such as road lines. If pose detection is enabled, pose information is added to pedestrian regions; if LiDAR is enabled, an intensity map is overlaid in the forward view. Background regions are also processed based on noise intensity. The weak, strong, and third control features are concatenated along the channel dimension to form a multi-channel control video.

[0114] In some embodiments, the weak control method may be Leres depth estimation; the strong control method may be Lineart standard detection; and the third control method may be Pidi line detection.

[0115] Control feature encoding: The control video is segmented and encoded using a variational autoencoder to generate latent control features, ensuring compatibility with the model input format.

[0116] Input video and mask preparation: The input videos for multiple views are stitched together in a uniform format, and a mask is generated for each view. If the target is T2V, the mask settings ensure the first frame retains the original content, while subsequent frames are generated by the model; if it is IT2V, all frames are generated by the model. Prepare positive and negative cue words.

[0117] Video Generation: Combining latent features, input video, mask, and cue words, a diffusion model generates a target sunny video. The generation process uses a specified scheduler (e.g., DDIM), sets the guiding scale (e.g., 6.0), and the number of inference steps (e.g., 20) to ensure high quality and consistency of the generated video.

[0118] This invention eliminates the need for a unified modeling system based on a bird's-eye view (BEV) coordinate system, effectively reducing the computational overhead and deployment complexity of geometric calibration and coordinate transformation. Furthermore, it eliminates the need for paired data between clear and inclement scenes during training, significantly reducing the cost of collecting and labeling high-quality samples and improving the model's usability in real-world scenarios. This invention supports simultaneous generation from up to seven perspectives, ensuring geometric and semantic consistency across perspectives. Through a strategy combining text-to-video (T2V) with subsequent image-to-video (I2V), it achieves stable video generation of unlimited duration, effectively avoiding jitter and dynamic target deformation in long-sequence generation. Moreover, this invention achieves the conversion of various inclement environments such as rain, snow, and fog into a unified clear style, maintaining the semantic structure of the original scene and ensuring the clarity and positional accuracy of key targets such as vehicles, pedestrians, and lane lines. This guarantees the consistency and accuracy of target detection (OD) and lane detection (LD), while also allowing users to flexibly adjust the generation intensity, achieving controllable enhancement of image clarity and realism.

[0119] It should be noted that the above figures are merely illustrative representations of the processes included in methods according to some embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0120] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0121] Figure 9 This is a block diagram illustrating a video processing apparatus according to some embodiments of the present disclosure. (Refer to...) Figure 9 The device includes: an acquisition module 901, a first encoding module 902, a second encoding module 903, a noise reduction guidance module 904, and a decoding module 905.

[0122] The acquisition module 901 is used to acquire the source video and the multimodal control conditions corresponding to the source video. The multimodal control conditions include depth information representing the geometric structure of the scene, contour information representing the contour of the target, and segmentation mask of the video frame. The first encoding module 902 is used to perform feature encoding on the source video to obtain the potential features to be denoised. The second encoding module 903 is used to perform feature encoding on the multimodal control conditions to obtain the control latent features; The denoising guidance module 904 is used to guide the latent features to be denoised by controlling the latent features, so as to predict and remove the noise in the latent features to be denoised, and obtain the target latent features. The decoding module 905 is used to decode the latent features of the target to obtain the target video, which has a higher visual quality than the source video.

[0123] In some embodiments of this disclosure, the denoising guidance module 904 includes an iterative denoising unit and an iterative control unit.

[0124] The iterative denoising unit is used to predict noise and remove the predicted noise based on the current latent features to be denoised and the control latent features, so as to obtain the updated latent features to be denoised. The iterative control unit is used to predict noise and remove predicted noise based on the updated latent features to be denoised and control latent features, until a preset number of iterations are completed or a preset stopping condition is met, to obtain the target latent features.

[0125] In some embodiments of this disclosure, the iterative denoising unit is used to concatenate or add the latent features to be denoised with the control latent features to obtain fused features; the fused features are input into the denoising network, which predicts the noise in the latent features to be denoised based on the guidance of the control latent features, removes the noise from the latent features to be denoised, and outputs the updated latent features to be denoised.

[0126] In some embodiments of this disclosure, the denoising network employs a diffusion model network that includes rotational position coding, which is used to encode the positional information of the fused features in the spatial and temporal dimensions.

[0127] In some embodiments of this disclosure, the multimodal control conditions also include radar information; the above-described apparatus also includes a text acquisition module and a text encoding module.

[0128] The text acquisition module is used to acquire text prompts as semantic guidance input; The text encoding module is used to embed and encode text prompts to obtain latent text features; The iterative denoising unit is used to predict noise for the current latent features to be denoised by using control latent features and text latent features together, and remove the predicted noise.

[0129] In some embodiments of this disclosure, the acquisition module 901 further includes a parameter acquisition unit.

[0130] The parameter acquisition unit is used to acquire the guiding scale and the number of inference steps. The guiding scale is used to control the degree to which the target video follows the multimodal control conditions, and the number of inference steps is used to control the number of denoising iterations.

[0131] In some embodiments of this disclosure, the second encoding module 903 is used to independently encode the depth information, contour information, and segmentation mask of the video frame to obtain the corresponding depth latent features, contour latent features, and segmentation latent features; and to perform feature fusion on the depth latent features, contour latent features, and segmentation latent features to obtain control latent features.

[0132] Regarding the video processing apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the video processing method, and will not be elaborated upon here.

[0133] This disclosure also provides a vehicle, which can be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicles. The vehicle can be a driver-assisted vehicle, a semi-driver-assisted vehicle, or a driver-free vehicle.

[0134] Vehicles can include various subsystems, such as infotainment systems, perception systems, decision control systems, drive systems, and computing platforms. A vehicle can also include more or fewer subsystems, and each subsystem can include multiple components. Furthermore, each subsystem and each component of the vehicle can be interconnected via wired or wireless means.

[0135] In some embodiments, an infotainment system may include a communication system, an entertainment system, and a navigation system, etc.

[0136] The perception system may include several types of sensors used to sense information about the environment surrounding the vehicle. For example, the perception system may include a global positioning system (which may be GPS, BeiDou, or other positioning systems), an inertial measurement unit (IMU), lidar, millimeter-wave radar, ultrasonic radar, and camera devices.

[0137] The decision control system may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0138] A drive system may include components that provide powered motion to a vehicle. In one embodiment, a drive system may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of internal combustion engines, electric motors, and compressed air engines. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0139] Some or all of the vehicle's functions are controlled by a computing platform. The computing platform may include at least one processor and memory, the processor being able to execute instructions stored in the memory.

[0140] The processor can be any conventional processor, such as a commercially available CPU. The processor can also include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems-on-chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0141] Memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0142] In addition to instructions, memory can also store data, such as road maps, route information, and vehicle position, direction, and speed. The data stored in memory can be used by the computing platform.

[0143] In this embodiment of the disclosure, the processor can execute instructions to complete all or part of the steps of the video processing method described above.

[0144] Figure 10 This is a block diagram illustrating an electronic device 1000 according to some embodiments of the present disclosure. For example, the electronic device 1000 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0145] Reference Figure 10 The electronic device 1000 may include one or more of the following components: a processing component 1002, a memory 1004, a power component 1006, a multimedia component 1008, an audio component 1010, an input / output (I / O) interface 1012, a sensor component 1014, and a communication component 1016.

[0146] Processing component 1002 typically controls the overall operation of electronic device 1000, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1002 may include one or more processors 1020 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1002 may include one or more modules to facilitate interaction between processing component 1002 and other components. For example, processing component 1002 may include a multimedia module to facilitate interaction between multimedia component 1008 and processing component 1002.

[0147] Memory 1004 is configured to store various types of data to support the operation of device 1000. Examples of this data include instructions for any application or method operating on electronic device 1000, contact data, phonebook data, messages, pictures, videos, etc. Memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0148] The power supply component 1006 provides power to various components of the electronic device 1000. The power supply component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1000.

[0149] Multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1008 includes a front-facing camera and / or a rear-facing camera. When the device 1000 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0150] Audio component 1010 is configured to output and / or input audio signals. For example, audio component 1010 includes a microphone (MIC) configured to receive external audio signals when electronic device 1000 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1004 or transmitted via communication component 1016. In some embodiments, audio component 1010 also includes a speaker for outputting audio signals.

[0151] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0152] Sensor assembly 1014 includes one or more sensors for providing state assessments of various aspects of electronic device 1000. For example, sensor assembly 1014 may detect the on / off state of device 1000, the relative positioning of components such as the display and keypad of electronic device 1000, changes in position of electronic device 1000 or a component of electronic device 1000, the presence or absence of user contact with electronic device 1000, the orientation or acceleration / deceleration of electronic device 1000, and temperature changes of electronic device 1000. Sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1014 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1014 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0153] Communication component 1016 is configured to facilitate wired or wireless communication between electronic device 1000 and other devices. Electronic device 1000 can access wireless networks based on communication standards, such as WiFi, 3G, 4G, 5G, other communication standards, or combinations thereof. In some embodiments of this disclosure, communication component 1016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of this disclosure, communication component 1016 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0154] In some embodiments of this disclosure, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the video processing method described above.

[0155] In some embodiments of this disclosure, a computer-readable storage medium including instructions is also provided, such as a memory including instructions that can be executed by the processor 1020 of the electronic device 1000 to perform the above-described method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0156] In some embodiments of this disclosure, computer instructions are stored on a computer-readable storage medium, which, when executed by a processor, implement the video processing method described in the above method embodiments.

[0157] In some embodiments of this disclosure, a computer-readable storage medium is one capable of sending, propagating, or transmitting computer instructions for use by or in connection with an instruction execution system, apparatus, or device. As an example, a computer-readable storage medium is a non-volatile storage medium.

[0158] In some embodiments of this disclosure, more specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, USB flash drives, portable hard drives, or any suitable combination of the foregoing.

[0159] In some embodiments of this disclosure, computational instructions contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0160] This disclosure also provides a computer program product storing instructions that, when executed by a computer, cause the computer to perform the video processing method described in the above-described method embodiments. These instructions may be program code. In specific implementations, the program code may be written using any combination of one or more programming languages. The program code may be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0161] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0162] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video processing method, characterized in that, include: Obtain the source video and the corresponding multimodal control conditions, wherein the multimodal control conditions include depth information representing the geometric structure of the scene, contour information representing the contour of the target, and segmentation mask of the video frame; The source video is feature-encoded to obtain the potential features to be denoised; The multimodal control conditions are feature-encoded to obtain the control latent features; The control latent features are used to guide the latent features to be denoised, so as to predict and remove noise in the latent features to be denoised, thereby obtaining the target latent features; The target's latent features are decoded to obtain a target video, the visual quality of which is stronger than that of the source video.

2. The method according to claim 1, characterized in that, The step of guiding the denoised latent features using the control latent features to predict and remove noise from the denoised latent features to obtain the target latent features includes: Based on the current potential features to be denoised and the control potential features, noise is predicted and the predicted noise is removed to obtain the updated potential features to be denoised. Based on the updated latent features to be denoised and the control latent features, noise prediction is performed and the predicted noise is removed until a preset number of iterations are completed or a preset stopping condition is met, thereby obtaining the target latent features.

3. The method according to claim 2, characterized in that, The process of predicting noise and removing predicted noise based on the current latent features to be denoised and the control latent features to obtain updated latent features to be denoised includes: The latent features to be denoised are concatenated or added to the latent control features to obtain fused features; The fused features are input into the denoising network, which predicts the noise in the potential features to be denoised based on the guidance of the control latent features, removes the noise from the potential features to be denoised, and outputs the updated potential features to be denoised.

4. The method according to claim 3, characterized in that, The denoising network employs a diffusion model network that includes rotational position encoding, which is used to encode the positional information of the fused features in the spatial and temporal dimensions.

5. The method according to claim 2, characterized in that, The multimodal control conditions also include radar information; the method further includes: Obtain text prompts as semantically guided input; The text prompt words are embedded and encoded to obtain the text latent features; The step of predicting noise and removing predicted noise based on the current potential features to be denoised and the control potential features includes: using the control potential features and the text potential features to jointly predict noise on the current potential features to be denoised, and removing the predicted noise.

6. The method according to claim 5, characterized in that, The method further includes: The guiding scale and inference steps are obtained. The guiding scale is used to control the degree to which the target video follows the multimodal control conditions, and the inference steps are used to control the number of denoising iterations.

7. The method according to claim 1, characterized in that, The step of feature encoding the multimodal control conditions to obtain latent control features includes: Independent feature encoding is performed on the depth information, the contour information, and the segmentation mask of the video frame to obtain the corresponding latent depth features, latent contour features, and latent segmentation features; The control latent features are obtained by fusing the depth latent features, the contour latent features, and the segmentation latent features.

8. A video processing apparatus, characterized in that, include: The acquisition module is used to acquire source video and multimodal control conditions corresponding to the source video, wherein the multimodal control conditions include depth information representing the geometric structure of the scene, contour information representing the contour of the target, and segmentation mask of the video frame; The first encoding module is used to perform feature encoding on the source video to obtain potential features to be denoised; The second encoding module is used to perform feature encoding on the multimodal control conditions to obtain potential control features; A denoising guidance module is used to guide the potential feature to be denoised using the control latent feature, so as to predict and remove noise in the potential feature to be denoised, and obtain the target latent feature; A decoding module is used to decode the latent features of the target to obtain a target video, wherein the visual quality of the target video is stronger than that of the source video.

9. The apparatus according to claim 8, characterized in that, The noise reduction guidance module includes: An iterative denoising unit is used to predict noise and remove the predicted noise based on the current potential features to be denoised and the control potential features, so as to obtain updated potential features to be denoised. An iterative control unit is used to predict noise and remove predicted noise based on the updated potential features to be denoised and the control potential features, until a preset number of iterations are completed or a preset stopping condition is met, to obtain the target potential features.

10. The apparatus according to claim 9, characterized in that, The multimodal control conditions also include radar information; the device also includes: The text acquisition module is used to acquire text prompts as semantic guidance input; The text encoding module is used to embed and encode the text prompt words to obtain latent text features; The iterative denoising unit is used to perform noise prediction on the current potential features to be denoised using the control latent features and the text latent features, and remove the predicted noise.

11. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the steps of the video processing method according to any one of claims 1 to 7.

12. A computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of a mobile terminal, enable the mobile terminal to perform the steps of the video processing method according to any one of claims 1 to 7.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the video processing method as described in any one of claims 1 to 7.

Images