Video extension processing method and device, storage medium and electronic equipment
By extracting the spatial and temporal reference features of video frames, fusion of features and inputting them into the neural network model, the problems of pixel jitter and local anomalies after video extension processing are solved, and the video stability and adaptive extension are achieved.
Patent Information
- Application Number
- CN202411328963.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2025-02-14
AI Technical Summary
In the prior art, there are problems such as pixel jitter, local abnormalities and blur after video extension processing.
By extracting the spatial reference features and temporal reference features of the video frame, the feature fusion process is performed, and input it into the neural network model to obtain the extended target video.
The stability, rationality and adaptability of video size extension are achieved, and problems such as pixel jitter, local damage and blur are avoided.
Smart Images
Figure CN119450105A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of deep learning and computer vision intelligence, and specifically to a method and device for processing video extension, a system for processing video extension, and the present application also relates to a computer storage medium and an electronic device. Background Art
[0002] With the development of deep learning and artificial intelligence technologies, they have also been integrated into data processing scenarios such as video and images, thereby improving processing effects and efficiency. For example, in order to adapt to the presentation of video data on various display platforms, the aspect ratio of existing videos needs to be extended to accommodate playback devices of different sizes. This process is called video outpainting technology.
[0003] The video extension technology in the prior art is mainly a data processing process of extending the boundary of the video through a diffusion model, so that the video can adapt to display devices of different sizes. Summary of the invention
[0004] The present application provides a video extension processing method to solve the pixel jitter problem and local abnormality problem that occur after the video is extended in the prior art.
[0005] The present application provides a video extension processing method, including:
[0006] According to the spatial reference information and the temporal reference information of the video data to be extended, performing spatial reference feature extraction and temporal reference feature extraction on the video frames in the video data to be extended, and determining a first spatial reference feature and a second spatial reference feature, as well as a first temporal reference feature and a second temporal reference feature;
[0007] Performing feature fusion processing on the first spatial reference feature, the second spatial reference feature, and the first time reference feature respectively, to determine a first fusion reference feature corresponding to the first spatial reference feature, a second fusion reference feature corresponding to the second spatial reference feature, and a third fusion reference feature corresponding to the first time reference feature;
[0008] The first fusion reference feature, the second fusion reference feature, the third fusion reference feature and the second time reference feature are input into a neural network model to obtain an extended target video.
[0009] In some embodiments, the extracting spatial reference features and extracting temporal reference features of video frames in the video data to be stretched according to the spatial reference information and temporal reference information of the video data to be stretched, and determining the first spatial reference feature and the second spatial reference feature, as well as the third temporal reference feature and the fourth temporal reference feature, comprises:
[0010] Extracting texture features and semantic features of the video frame according to the spatial reference information;
[0011] Determining the texture feature as the first spatial reference feature;
[0012] Determining the semantic feature as the second spatial reference feature;
[0013] Extracting motion features and optical flow features of the video frame according to the time reference information;
[0014] determining the motion feature as the first time reference feature;
[0015] The optical flow feature is determined as the second time reference feature.
[0016] In some embodiments, extracting texture features of the video frame according to the spatial reference information includes:
[0017] The video frame in the video data to be extended is input into a first network model to extract the texture feature and obtain the texture feature.
[0018] In some embodiments, extracting the semantic features of the video frame according to the spatial reference information includes:
[0019] Inputting the video frame in the video data to be extended into a second network model to obtain first semantic information;
[0020] extracting a first semantic feature according to the first semantic information;
[0021] Inputting the video frame in the video data to be extended into a third network model to obtain second semantic information;
[0022] A second semantic feature is extracted according to the second semantic information.
[0023] In some embodiments, extracting a first semantic feature according to the first semantic information includes:
[0024] Inputting the first semantic information into a fourth network model and / or a fifth network model to perform feature extraction to obtain the first semantic feature;
[0025] The extracting a second semantic feature according to the second semantic information includes:
[0026] The second semantic information is input into the fourth network model and / or the fifth network model to perform feature extraction to obtain the second semantic feature.
[0027] In some embodiments, extracting the motion feature of the video frame according to the time reference information includes:
[0028] The video frames in the to-be-extended video data are input into a fourth network model to extract motion features, thereby obtaining the motion features.
[0029] In some embodiments, extracting the optical flow features of the video frame according to the time reference information includes:
[0030] Extracting optical flow information according to the video frame in the video data to be extended to obtain optical flow information;
[0031] The optical flow information is input into the fifth network model to extract optical flow features and obtain the optical flow features.
[0032] In some embodiments, the performing feature fusion processing on the first spatial reference feature, the second spatial reference feature, and the first time reference feature respectively to determine a first fused reference feature corresponding to the first spatial reference feature, a second fused reference feature corresponding to the second spatial reference feature, and a third fused reference feature corresponding to the first time reference feature includes:
[0033] Perform feature splicing on the texture feature and latent space noise data to generate the first fusion reference feature;
[0034] Fusing the semantic features through a cross-attention mechanism to generate the second fused reference features;
[0035] The motion features are fused through the cross-attention mechanism to generate the third fused reference features.
[0036] In some embodiments, inputting the first fusion reference feature, the second fusion reference feature, the third fusion reference feature, and the second time reference feature into a neural network to obtain an extended target video includes:
[0037] The first fusion reference feature, the second fusion reference feature, the third fusion reference feature and the second time reference feature are input into a diffusion model of a neural network to perform inverse diffusion to obtain an extended target video.
[0038] The present application also provides a video extension processing device, including:
[0039] A first determining unit is used to perform spatial reference feature extraction and temporal reference feature extraction on video frames in the video data to be extended according to the spatial reference information and temporal reference information of the video data to be extended, and determine a first spatial reference feature and a second spatial reference feature, as well as a first temporal reference feature and a second temporal reference feature;
[0040] a second determining unit, configured to perform feature fusion processing on the first spatial reference feature, the second spatial reference feature, the first time reference feature, and the second time reference feature, respectively, to determine a first fusion reference feature corresponding to the first spatial reference feature, a second fusion reference feature corresponding to the second spatial reference feature, and a third fusion reference feature corresponding to the first time reference feature;
[0041] An acquisition unit is used to input the first fusion reference feature, the second fusion reference feature, the third fusion reference feature and the second time reference feature into a neural network model to acquire an extended target video.
[0042] The present application also provides a video extension processing system, including: a spatial reference module, a temporal reference module and a deep learning module;
[0043] The spatial reference module is used to extract spatial reference features from the video frames in the extended video data to determine a first spatial reference feature and a second spatial reference feature;
[0044] The time reference module is used to extract time reference features from the video frames in the video data to be extended, and determine a first time reference feature and a second time reference feature;
[0045] The deep learning module is used to perform feature fusion processing on the first spatial reference feature, the second spatial reference feature and the first time reference feature respectively, and determine the first fusion reference feature corresponding to the first spatial reference feature, the second fusion reference feature corresponding to the second spatial reference feature, and the third fusion reference feature corresponding to the first time reference feature, which are used as input data of the neural network model for learning to obtain the extended target video.
[0046] The present application also provides an electronic device, comprising:
[0047] processor;
[0048] The memory is used to store a program for processing data generated by the electronic device. When the program is read and executed by the processor, the processing method of video extension as described above is executed.
[0049] The present application also provides a computer storage medium, characterized in that it includes a computer program, and when the computer program is run on an electronic device, the electronic device executes the video extension processing method as described above.
[0050] Compared with the prior art, this application has the following advantages:
[0051] The video extension processing method provided in the present application not only involves the spatial characteristics of the video data but also the temporal characteristics of the video data. The specific spatiotemporal reference information of the video itself is used as the input reference feature data of the diffusion model to realize the processing of video size extension, ensure the stability, rationality and adaptability of the expanded video, and avoid any one or more problems such as pixel jitter, local damage, blur, etc. in the video after extension processing in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 This is a flow chart of a video extension processing method provided by this application;
[0053] Figure 2 It is a structural schematic diagram of a video extension processing device provided by the present application;
[0054] Figure 3 It is a structural diagram of a video extension processing system provided by the present application;
[0055] Figure 4 This is a schematic diagram of the execution flow architecture of a video extension processing system provided by the present application;
[0056] Figure 5 It is a structural schematic diagram of an electronic device provided by this application. DETAILED DESCRIPTION
[0057] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.
[0058] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The descriptions used in this application and the appended claims, such as "a", "first", and "second", are not limitations on quantity or sequence, but are used to distinguish the same type of information from each other.
[0059] Based on the above background technology, it can be known that the technical concept of this application comes from video extension technology, that is, how to process video data so that it can adapt to display devices of different sizes. In terms of application scenarios, video, as a method of information display, has become an indispensable part of life. Secondly, with the development of live broadcast application scenarios, technologies such as video playback and slicing have also emerged. Whether it is the playback scene of live video, the playback scene of video slicing, or the original video playback scene, it is necessary to adapt the specific size of the video to different playback devices. The existing videos in the prior art have a specific aspect ratio (such as 16:9, 4:3, 1:1, etc.), but in order to make the video adapt to playback devices with different aspect ratios, it is necessary to spatially extend the video, that is, to achieve the size extension of the video in the vertical, horizontal and surrounding directions. In the past two years, the generation of visual content based on the diffusion model has developed rapidly, and the spatial extension of the video can be achieved based on the technology of the diffusion model. However, the video after extension processing in the prior art has pixel jitter problems, as well as local damage and blurring of the video.
[0060] In view of this, the present application provides a method for processing video extension, which can avoid problems such as video jitter, local damage, blur, etc. after video extension.
[0061] like Figure 1 As shown, Figure 1 : is a flowchart of a video extension processing method provided by the present application, the method comprising:
[0062] Step S101: performing spatial reference feature extraction and temporal reference feature extraction on video frames in the video data to be extended according to the spatial reference information and temporal reference information of the video data to be extended, and determining a first spatial reference feature and a second spatial reference feature, as well as a first temporal reference feature and a second temporal reference feature;
[0063] Step S102: performing feature fusion processing on the first spatial reference feature, the second spatial reference feature, and the first time reference feature respectively, to determine a first fusion reference feature corresponding to the first spatial reference feature, a second fusion reference feature corresponding to the second spatial reference feature, and a third fusion reference feature corresponding to the first time reference feature;
[0064] Step S103: input the first fusion reference feature, the second fusion reference feature, the third fusion reference feature and the second time reference feature into a neural network model to obtain an extended target video.
[0065] It should be noted that the implementation process of the above steps S101 to S103 of the present application can be to implement the extended processing of video data based on a deep learning network model, such as a diffusion model, but it is not limited to a diffusion model. In the face of a large number of existing deep learning network models, any network model that can implement the above steps of the present application is acceptable. In this embodiment, only the diffusion model is used as an example to illustrate the specific implementation process, which is not limited to the diffusion model. In addition, the network model is also a model trained based on a large amount of sample data. The same network model structure can be used in the training stage and the application scenario. Of course, the network model can also be modified and adjusted according to different application scenarios and requirements. For example, when the application scenario is relatively simple, the network model can remove or reduce unnecessary modules or parameters for implementing the corresponding functions. Before describing the above steps S101 to S103 in detail, a preliminary summary of the relevant technical terms involved in this embodiment is given.
[0066] Diffusion Model: It is a generative model based on probability theory. In computer vision, it trains neural networks by learning the inverse diffusion process to denoise Gaussian noise images and generate high-definition images.
[0067] 3D-UNet: The UNet network structure is a typical U-shaped structure that includes a downsampling module and an upsampling module. 3D-UNet is modified based on UNet so that the network can support the input of three-dimensional information.
[0068] 3D Reference Network (3D-RefNet): It uses the network structure of 3D-Net, plays the role of reference information feature extraction, and assists 3D-UNet in content generation.
[0069] Motion module: Video data in this module enables pixels to perform cross-temporal attention mechanisms and perceive temporal operation information.
[0070] ControlNet: In the Diffusion Model-based generation solution, the control network can control the generated content results in a fine-grained manner.
[0071] Video extension: Reasonably fill the content at the edge of the video to expand the video size so that the video can adapt to playback devices and playback occasions with different aspect ratios.
[0072] OpenAI CLIP: CLIP (Contrastive Language–Image Pretraining: Multimodal Pretraining Neural Network Model) is a multimodal learning model developed by OpenAI, which aims to embed images and text into a shared semantic space through contrastive learning, that is, to match images and text. It achieves cross-modal understanding and processing of multiple tasks, such as image classification, text classification, image retrieval, etc. OpenAI CLIP is a neural network released by OpenAI that connects text and images.
[0073] Open CLIP generally refers to open source implementations or similar projects based on the OpenAI CLIP model.
[0074] Variational Autoencoder (VAE): An autoencoder network model based on variational inference. With the rapid development of deep learning and artificial intelligence, models such as autoencoders (AE) and variational autoencoders (VAE) have become key tools in understanding and generating class models. VAE can also be understood as a Bayesian neural network, that is, when the input is determined, the output obeys the probability distribution based on the input conditions. VAE directly maps the px(x) distribution to the known pz(z) distribution by learning the Encoder, and learns the Decoder so that the generative model can be mapped from the known pz(z) distribution to the px(x) distribution.
[0075] The specific implementation process of the above steps S101 to S103 is described below. In the process of understanding the above steps S101-S103, you can also combine Figure 3 and Figure 4 Content shown.
[0076] Regarding step S101: according to the spatial reference information and the temporal reference information of the video data to be extended, the spatial reference feature extraction and the temporal reference feature extraction are performed on the video frames in the video data to be extended to determine the first spatial reference feature and the second spatial reference feature, as well as the first temporal reference feature and the second temporal reference feature.
[0077] The purpose of step S101 is to extract feature data in the video to be extended from two aspects, namely, extracting spatial reference features (namely, the first spatial reference feature and the second spatial reference feature) from the perspective of spatial reference information, and extracting temporal reference features (namely, the first temporal reference feature and the second temporal reference feature) from the perspective of temporal reference information. The spatial reference features are correlated with the spatial data of the video frames in the video to be extended, and the temporal reference features are correlated with the temporal data (also understood as time series data) of the video frames in the video to be extended. The reference features may also be time series reference features.
[0078] For video data, it includes not only spatial information but also temporal information, that is, spatial information (SI, Spatial perceptual Information, also known as spatial complexity) and temporal information (TI, Temporal perceptual Information, also known as temporal complexity) can characterize the characteristics or features of the video. Generally, SI characterizes the amount of spatial detail in a frame of an image. The more complex the scene is in space, the higher the SI value. TI characterizes the temporal variation of the video sequence. Sequences with a higher degree of motion usually have higher TI values. In this embodiment, the feature data of the video to be extended are extracted from the two perspectives of space and time.
[0079] The specific implementation of the process of extracting the first spatial reference feature and the second spatial reference feature related to the space may include:
[0080] Step S101-11: extracting texture features and semantic features of the video frame according to the spatial reference information;
[0081] Step S101-12: determining the texture feature as the first spatial reference feature;
[0082] Step S101 - 13: Determine the semantic feature as the second spatial reference feature.
[0083] In this embodiment, in step S101-11, the texture features of the video frame are extracted according to the spatial reference information. The texture features of the video frame mainly refer to the texture information of the video frame image, which is a visual feature that does not depend on the color and brightness of the image, but is described by the local pattern and structure of the image. Texture features can provide information about the surface properties of the image, such as roughness, directionality and regularity. There are many methods for extracting texture features, and the common ones include grayscale co-occurrence matrix and feature extraction based on wavelet transform. The grayscale co-occurrence matrix converts the grayscale value of the image into texture features by calculating the grayscale value of the image, while the wavelet transform is a local transformation of time and frequency, which has the characteristics of multi-resolution analysis and can effectively extract the texture information of the image. In this embodiment, the extraction of texture features (first reference features) is realized by adopting the VEA network model. Of course, in addition to the above extraction method, it can also be an adversarial network model (Generative Adversarial Nets: GAN). The GAN network model can effectively extract the texture features of the video frame image through the competition mechanism of its generator and discriminator. The texture features of the video frame image can also be extracted by a Gabor filter. Therefore, in this embodiment, VEA is mainly used as an example for explanation, but it is not limited to the method used when extracting texture features. Therefore, the step S101-11 includes:
[0084] Step S101-11-1: input the video frame in the video data to be extended into the first network model to extract the texture features and obtain the texture features.
[0085] The first network model can be any one of the models such as VEA, GAN, Gabor, gray level co-occurrence matrix, etc. The first reference feature is a spatial feature of a video frame image extracted based on spatial reference information, and is not limited to the texture feature given in this embodiment, and can also include: spatial position features, shape features, size features, etc., which can mark feature data of element spatial information in the video frame image. The texture feature can be used to reduce the complexity of encoding, reduce the amount of calculation, and thus reduce the encoding time. In addition, the texture feature can also be used to improve the video frame image reconstruction performance, which helps to improve the effect and efficiency of video processing.
[0086] In this embodiment, semantic features, namely, second spatial reference features, can be obtained by analyzing the video frame images. In order to improve the accuracy of understanding the video frame images, the extraction of semantic features in step S101-11 can be performed at least twice through different network models. In this embodiment, the extraction of semantic features can be achieved by adopting a multimodal pre-trained neural network model, for example: extracting the first semantic information through OpenAI CLIP, and extracting the second semantic information through Open CLIP.
[0087] Therefore, in step S101-11, the semantic features of the video frame are extracted according to the spatial reference information, and the specific implementation process may include:
[0088] Step S101-11-21: inputting the video frame in the video data to be extended into the second network model to obtain first semantic information;
[0089] Step S101-11-22: extracting a first semantic feature according to the first semantic information; the second network model may be OpenAI CLIP.
[0090] Step S101-11-23: inputting the video frame in the video data to be extended into a third network model to obtain second semantic information;
[0091] Step S101-11-24: extracting a second semantic feature according to the second semantic information; the third network model may adopt Open CLIP.
[0092] It is understandable that in order to improve the efficiency of semantic feature extraction, the present embodiment adopts the CLIP-related artificial intelligence large model, namely OpenAI CLIP and Open CLIP to extract the first semantic information and the second semantic information respectively. In fact, the extraction of semantic features is not limited to the above CLIP model, for example: Word2Vec, GloVe, BERT, etc. or convolutional neural network models, etc.
[0093] The specific implementation process of step S101-11-22 may be: inputting the first semantic information into the fourth network model and / or the fifth network model to perform feature extraction to obtain the first semantic feature.
[0094] The specific implementation process of step S101-11-24 may be: inputting the second semantic information into the fourth network model and / or the fifth network model for feature extraction to obtain the second semantic feature.
[0095] Among them, the fourth network model and the fifth network model can be 3D-RefNet and 3D-UNet, which will be described later and are only briefly introduced here.
[0096] The above is a process of extracting the first spatial reference feature and the second spatial reference feature related to space from the perspective of spatial reference information for the video frame of the video data to be extended.
[0097] In step S101, it is necessary to extract not only the first spatial reference feature and the second spatial reference feature related to space, but also the first time reference feature and the second time reference feature related to time. Specifically, it may include:
[0098] Step S101-21: extracting motion features and optical flow features of the video frame according to the time reference information;
[0099] Step S101-22: determining the motion feature as the first time reference feature;
[0100] Step S101 - 23: Determine the optical flow feature as the second time reference feature.
[0101] The time reference information can be understood as the data of the video frame being able to represent information related to time or timing. The motion features and optical flow features related to the time reference information are both features of the video frame, wherein the optical flow features are used to characterize the motion information of the element objects in the video frame between consecutive frames. The motion features generally refer to the relevant information about the motion pattern and dynamic changes of the element objects in the video frame in the video sequence. The motion features and the optical flow features can be extracted using different network models respectively, therefore, the specific implementation process of the motion feature extraction in step S101-21 can include:
[0102] Step S101-211: input the video frame in the video data to be extended into the fourth network model to extract motion features and obtain the motion features.
[0103] The fourth network model may be a 3D reference network, i.e., 3D-RefNet, which uses the network structure of 3D-Net to extract relevant data of motion features from the three-dimensional space of the video frame. In this embodiment, the motion feature extraction may be performed by interpolating the video frame as the input of the 3D reference network to extract the motion features of the video frame.
[0104] The specific implementation process of optical flow feature extraction in step S101-21 may include:
[0105] Step S101-212: extracting optical flow information according to the video frame in the video data to be extended to obtain optical flow information;
[0106] Step S101-213: Input the optical flow information into the fifth network model to extract optical flow features and obtain the optical flow features.
[0107] The fifth network model may be a control network model (ControlNet), which can control the generated content in a fine-grained manner in a DiffusionModel generation scheme; ControlNet is a powerful neural network architecture that enhances the capabilities of large pre-trained text-to-image diffusion models by adding spatial condition control. The fifth network model may be integrated into the fourth network model or exist independently. In this embodiment, the integration approach can ensure the consistency of feature extraction.
[0108] In this embodiment, binary mask enhancement data can be used when extracting optical flow features through the control network model, that is, by randomly blocking or selectively focusing on specific areas of the video frame, the robustness and generalization ability of the control network model can be increased, while also reducing the amount of calculation, thereby reducing costs.
[0109] Regarding step S102: perform feature fusion processing on the first spatial reference feature, the second spatial reference feature, and the first time reference feature respectively to determine a first fusion reference feature corresponding to the first spatial reference feature, a second fusion reference feature corresponding to the second spatial reference feature, and a third fusion reference feature corresponding to the first time reference feature.
[0110] The purpose of this step is to perform feature processing such as fusion and splicing on the extracted reference features to generate corresponding fusion features.
[0111] It is understandable that the video data to be extended includes a plurality of time-sequential video frame images, and the time-sequential video frame images constitute complete video data. In this embodiment, the extension task of the video data to be extended can be implemented by a diffusion model.
[0112] It should be noted that the network models involved in this embodiment are all network models that have been trained for the processing tasks to be implemented.
[0113] In this embodiment, the diffusion model mainly adopts a neural network model based on 3D-UNet. Before the extended video is scaled up by the diffusion model, the above-mentioned reference features need to be fused. The specific implementation process may include:
[0114] Step S102-11: performing feature splicing on the texture feature and the latent space noise data to generate the first fusion reference feature;
[0115] Step S102-12: fusing the semantic features through a cross-attention mechanism to generate the second fused reference features;
[0116] Step S102-13: Fusing the motion features through the cross-attention mechanism to generate the third fused reference features.
[0117] The step S102-11 is to process the texture features and use the processed features as the input data of the diffusion model 3D-UNet. Because the diffusion model is a generative model, its principle is to gradually transform random noise into target data distribution by simulating the random diffusion process, thereby generating new data. In the forward processing process, the model gradually "diffuses" or "corrupts" the information in the video frame data (such as the video frame image) and finally converts it into noise. This process is usually achieved by gradually adding Gaussian noise to the data, which can be done by adding Gaussian noise to the latent space of the diffusion model until the entire video frame data is completely converted into random noise. In the reverse processing process, the model will reconstruct data from the video frame data with noise, thereby achieving the expansion of the video frame size. Therefore, in this embodiment, the extracted texture features, latent space noise data, and video frame mask data are feature spliced so that the texture features and noise data form forward feature data, that is, the first fusion reference feature, as an input data of the 3D-UNet network model.
[0118] The step S102-12 is to process the semantic features using a cross-attention mechanism, and use the processed features as input data of the 3D-UNet network model. Because the video data to be extended has a certain temporal sequence, in order to better maintain temporal consistency, the present embodiment fuses the first semantic feature and the second semantic feature with the motion feature through a cross-attention mechanism, so that when the video frame image is extended, the global video data information can be perceived, so that when there is spatial information overlap, the diffusion model can be more reasonably filled, thereby improving the rationality and adaptability of the extension. In the present embodiment, there is complementarity between the first semantic feature and the second voice feature, so that the semantic reference information is more comprehensive and complete, and more abundant reference data is provided for the implementation of the extension. That is to say, the first semantic information and the second semantic information are input into the fourth network model and / or the fifth network model to extract semantic features to obtain the first voice feature and the second semantic feature, and the first semantic feature and the second semantic feature are fused by the cross-attention mechanism, and the fused semantic feature is used as one of the input data of the fifth network model (3D-UNet).
[0119] Based on the above steps S102-11 to S102-13, the first fusion reference feature, the second fusion reference feature and the third fusion reference feature can be generated respectively, and these fusion reference features and the second time reference feature (optical flow feature) are used as input data of 3D-UNet in the diffusion model for expansion processing.
[0120] Regarding step S103: input the first fusion reference feature, the second fusion reference feature, and the third fusion reference feature into the neural network to obtain the extended target video.
[0121] The first fusion reference feature, the second fusion reference feature and the third fusion reference feature are input as input data and the second time reference feature into the 3D-UNet of the diffusion model. The 3D-Unet performs fusion processing on the input data and performs inverse diffusion on the fused data, that is, denoising the noise in the fused data, generating the extended target video data, and then outputting it. The size of the video extension processing can adjust the parameters of the diffusion model in real time according to the application scenario. The diffusion model is a trained model and can adjust parameters according to different requirements to adapt to the output requirements of different display devices.
[0122] The above is a description of a video extension processing method provided by the present application using a diffusion model as an example. It can be understood that the video extension processing method provided by the present application not only involves the spatial characteristics of the video data but also the temporal characteristics of the video data. The specific spatiotemporal reference information of the video itself is used as the input reference feature data of the diffusion model to realize the processing of video size extension, ensure the stability, rationality and adaptability of the expanded video, and avoid any problems such as pixel jitter, local damage, blur, etc. in the video after extension processing in the prior art.
[0123] It can be understood that the models involved in the above-mentioned video extension processing method can be pre-trained using a large amount of sample data. The training process is as detailed above. The model is optimized by adjusting the loss function loss to ensure the stability of video extension during actual video extension processing.
[0124] It should be noted that the video extension processing method provided in this application can be applied to any device and scenario such as a video publisher, video display device, video editor, video clip cutter, etc., and can be processed in batches to improve processing efficiency.
[0125] The above is a detailed description of an embodiment of a method for processing video extension provided by the present application. Corresponding to the embodiment of a method for processing video extension provided above, the present application also discloses an embodiment of a device for processing video extension. Figure 2 Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.
[0126] like Figure 2 As shown, Figure 2: is a structural schematic diagram of a video extension processing device provided by the present application, the device comprising:
[0127] A first determining unit 201 is used to perform spatial reference feature extraction and temporal reference feature extraction on video frames in the video data to be extended according to the spatial reference information and temporal reference information of the video data to be extended, and determine a first spatial reference feature and a second spatial reference feature, as well as a first temporal reference feature and a second temporal reference feature;
[0128] The second determining unit 202 is used to perform feature fusion processing on the first spatial reference feature, the second spatial reference feature, the first time reference feature and the second time reference feature respectively, to determine a first fusion reference feature corresponding to the first spatial reference feature, a second fusion reference feature corresponding to the second spatial reference feature, and a third fusion reference feature corresponding to the first time reference feature;
[0129] The acquisition unit 203 is used to input the first fusion reference feature, the second fusion reference feature, the third fusion reference feature and the second time reference feature into the neural network model to obtain the extended target video.
[0130] In this embodiment, the first determining unit 201 includes: a first extracting subunit, a first determining subunit, a second determining subunit, a second extracting subunit, a third determining subunit, and a fourth determining subunit.
[0131] The first extraction subunit is used to extract texture features and semantic features of the video frame according to the spatial reference information; specifically, it is used to input the video frame in the video data to be extended into the first network model to extract the texture features and obtain the texture features; and it is used to input the video frame in the video data to be extended into the second network model to obtain first semantic information, and extract first semantic features according to the first semantic information; specifically, it can be inputting the first semantic information into the fourth network model and / or the fifth network model to extract features and obtain the first semantic features; and it is used to input the video frame in the video data to be extended into the third network model to obtain second semantic information, and extract second semantic features according to the second semantic information, specifically, it can be inputting the second semantic information into the fourth network model and / or the fifth network model to extract features and obtain the second semantic features.
[0132] The first determining subunit is used to determine the texture feature extracted by the first extracting subunit as the first spatial reference feature.
[0133] The second determining subunit is used to determine the semantic feature extracted by the first extracting subunit as the second spatial reference feature.
[0134] The second extraction subunit is used to extract the motion features and optical flow features of the video frame according to the time reference information; specifically, it is used to input the video frame in the video data to be extended into the fourth network model to extract the motion features and obtain the motion features; and it is used to extract the optical flow information according to the video frame in the video data to be extended and obtain the optical flow information; and input the optical flow information into the fifth network model to extract the optical flow features and obtain the optical flow features.
[0135] The third determining subunit is used to determine the motion feature extracted by the second extracting subunit as the first time reference feature.
[0136] The fourth determining subunit is used to determine the optical flow feature extracted by the second extracting subunit as the second time reference feature.
[0137] For the specific implementation process of the first determining unit 201, reference may be made to the relevant contents in the above method embodiment, which will not be described in detail here.
[0138] In this embodiment, the second determining unit 202 includes: a first generating subunit, a second generating subunit and a third generating subunit.
[0139] The first generating subunit is used to perform feature splicing on the texture feature and the latent space noise data to generate the first fusion reference feature;
[0140] The second generating subunit is used to perform feature fusion on the semantic features through a cross attention mechanism to generate the second fused reference features;
[0141] The third generating subunit is used to perform feature fusion on the motion features through the cross attention mechanism to generate the third fused reference features.
[0142] For the specific implementation process of the second determining unit 202, reference may be made to the relevant contents in the above method embodiment, which will not be described in detail here.
[0143] In this embodiment, the acquisition unit 203 includes: a fusion processing subunit and a generation subunit.
[0144] The fusion processing subunit is used to input the first fusion reference feature, the second fusion reference feature, the third fusion reference feature and the second time reference feature as input data into 3D-UNet for fusion processing.
[0145] The generating subunit is used to perform inverse diffusion on the fused data in the diffusion model of 3D-UNet to generate an extended target video.
[0146] For the specific implementation process of the acquisition unit 203, reference may be made to the relevant contents in the above method embodiment, which will not be described in detail here.
[0147] The above is a description of an embodiment of a video extension processing device provided in the present application. For the specific content of the device embodiment, please refer to the content of the above-mentioned method embodiment, which will not be repeated here.
[0148] Based on the above, the present application also provides a video extension processing system, such as Figure 3 and Figure 4 As shown, Figure 3 is a structural diagram of a video extension processing system provided by the present application, Figure 4 30 is a schematic diagram of the execution flow architecture of a video extension processing system provided by the present application. The processing system includes: a spatial reference module 301, a temporal reference module 302 and a deep learning module 303.
[0149] The spatial reference module 301 is used to extract spatial reference features of video frames in the extended video data, and determine the first spatial reference features and the second spatial reference features corresponding to the spatial reference information; in this embodiment, the spatial reference module extracts the first spatial reference features in the extended video through the VAE network model, that is, the texture features. The second spatial reference features in the extended video are extracted through the CLIP network model. In this embodiment, the first semantic information can be obtained through the OpenAI CLIP network model, the second semantic information can be obtained through OPenCLIP, and the semantic features can be extracted through the fourth network model and / or the fifth network model.
[0150] The temporal reference module 302 is used to extract temporal reference features from the video frames in the video data to be extended, and determine the first temporal reference features and the second temporal reference features corresponding to the temporal reference information; in this embodiment, the temporal reference module extracts the first temporal reference features in the video to be extended through a 3D reference network model, i.e., the motion features, and extracts the second temporal reference features in the video to be extended through a control network model (ControNet), i.e., the optical flow features.
[0151] The deep learning module 303 is used to perform feature fusion processing on the first spatial reference feature, the second spatial reference feature, and the first time reference feature, respectively, and determine the first fusion reference feature corresponding to the first spatial reference feature, the second fusion reference feature corresponding to the second spatial reference feature, the third fusion reference feature corresponding to the first time reference feature, and the second time reference feature as input data for learning to obtain the extended target video. In this embodiment, the deep learning module adopts a diffusion model, and the backbone network is 3D-UNet. The first fusion reference feature, the second fusion reference feature, the third fusion reference feature, and the second time reference feature are used as input data and input into the 3D-UNet of the diffusion model. The 3D-Unet performs fusion processing on the input data, and performs inverse diffusion on the fused data, that is, denoising the noise in the fused data, generating the extended target video data, and then outputting it. The extended processing size of the video to be extended can adjust the parameters of the diffusion model in real time according to the application scenario. The diffusion model is a trained model, and can adjust parameters in combination with different requirements to adapt to the output requirements of different display devices.
[0152] For the specific content of the above-mentioned processing system, please refer to the content of the above-mentioned processing method, which will not be described in detail here.
[0153] Based on the above content, the present application also provides an electronic device, such as Figure 5 As shown, the electronic device includes: a processor 501 and a memory 502, wherein the memory 502 is used to store a program for processing data generated by the electronic device, and when the program is read and executed by the processor, the processing method for video extension as described above is executed. In this embodiment, the electronic device may be a video publisher, etc.
[0154] Based on the above content, the present application also provides a computer storage medium, including a computer program, when the computer program is run on an electronic device, the electronic device executes the relevant content of the video extension processing method as described above.
[0155] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0156] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0157] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0158] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.
[0159] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0160] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.
Claims
1. A method for processing video extension, characterized in that: include: According to the spatial reference information and the temporal reference information of the video data to be extended, performing spatial reference feature extraction and temporal reference feature extraction on the video frames in the video data to be extended, and determining a first spatial reference feature and a second spatial reference feature, as well as a first temporal reference feature and a second temporal reference feature; Performing feature fusion processing on the first spatial reference feature, the second spatial reference feature, and the first time reference feature respectively, to determine a first fusion reference feature corresponding to the first spatial reference feature, a second fusion reference feature corresponding to the second spatial reference feature, and a third fusion reference feature corresponding to the first time reference feature; The first fusion reference feature, the second fusion reference feature, the third fusion reference feature and the second time reference feature are input into a neural network model to obtain an extended target video.
2. The video extension processing method according to claim 1, characterized in that: The method of performing spatial reference feature extraction and time reference feature extraction on video frames in the video data to be extended according to the spatial reference information and time reference information of the video data to be extended, and determining a first spatial reference feature and a second spatial reference feature, as well as a third time reference feature and a fourth time reference feature, includes: Extracting texture features and semantic features of the video frame according to the spatial reference information; Determining the texture feature as the first spatial reference feature; Determining the semantic feature as the second spatial reference feature; Extracting motion features and optical flow features of the video frame according to the time reference information; determining the motion feature as the first time reference feature; The optical flow feature is determined as the second time reference feature.
3. The video extension processing method according to claim 2, characterized in that: The extracting the texture feature of the video frame according to the spatial reference information includes: The video frame in the video data to be extended is input into a first network model to extract the texture feature and obtain the texture feature.
4. The video extension processing method according to claim 2, characterized in that: The extracting the semantic features of the video frame according to the spatial reference information includes: Inputting the video frame in the video data to be extended into a second network model to obtain first semantic information; extracting a first semantic feature according to the first semantic information; Inputting the video frame in the video data to be extended into a third network model to obtain second semantic information; A second semantic feature is extracted according to the second semantic information.
5. The video extension processing method according to claim 4, characterized in that: The extracting a first semantic feature according to the first semantic information includes: Inputting the first semantic information into a fourth network model and / or a fifth network model to perform feature extraction to obtain the first semantic feature; The extracting a second semantic feature according to the second semantic information includes: The second semantic information is input into the fourth network model and / or the fifth network model to perform feature extraction to obtain the second semantic feature.
6. The video extension processing method according to claim 2, characterized in that: The extracting the motion feature of the video frame according to the time reference information includes: The video frames in the to-be-extended video data are input into a fourth network model to extract motion features, thereby obtaining the motion features.
7. The video extension processing method according to claim 2, characterized in that: The extracting the optical flow feature of the video frame according to the time reference information includes: Extracting optical flow information according to the video frame in the video data to be extended to obtain optical flow information; The optical flow information is input into the fifth network model to extract optical flow features and obtain the optical flow features.
8. The video extension processing method according to claim 2, characterized in that: The performing feature fusion processing on the first spatial reference feature, the second spatial reference feature, and the first time reference feature respectively to determine a first fusion reference feature corresponding to the first spatial reference feature, a second fusion reference feature corresponding to the second spatial reference feature, and a third fusion reference feature corresponding to the first time reference feature, includes: Perform feature splicing on the texture feature and latent space noise data to generate the first fusion reference feature; Fusing the semantic features through a cross-attention mechanism to generate the second fused reference features; The motion features are fused through the cross-attention mechanism to generate the third fused reference features.
9. The video extension processing method according to claim 2, characterized in that: The step of inputting the first fusion reference feature, the second fusion reference feature, the third fusion reference feature, and the second time reference feature into a neural network to obtain an extended target video includes: The first fusion reference feature, the second fusion reference feature, the third fusion reference feature and the second time reference feature are input into a diffusion model of a neural network to perform inverse diffusion to obtain an extended target video.
10. A video extension processing device, characterized in that: include: A first determining unit is used to perform spatial reference feature extraction and temporal reference feature extraction on video frames in the video data to be extended according to the spatial reference information and temporal reference information of the video data to be extended, and determine a first spatial reference feature and a second spatial reference feature, as well as a first temporal reference feature and a second temporal reference feature; a second determining unit, configured to perform feature fusion processing on the first spatial reference feature, the second spatial reference feature, the first time reference feature, and the second time reference feature, respectively, to determine a first fusion reference feature corresponding to the first spatial reference feature, a second fusion reference feature corresponding to the second spatial reference feature, and a third fusion reference feature corresponding to the first time reference feature; An acquisition unit is used to input the first fusion reference feature, the second fusion reference feature, the third fusion reference feature and the second time reference feature into a neural network model to acquire an extended target video.
11. A video extension processing system, characterized in that: include: Spatial reference module, temporal reference module and deep learning module; The spatial reference module is used to extract spatial reference features from the video frames in the extended video data to determine a first spatial reference feature and a second spatial reference feature; The time reference module is used to extract time reference features from the video frames in the video data to be extended, and determine a first time reference feature and a second time reference feature; The deep learning module is used to perform feature fusion processing on the first spatial reference feature, the second spatial reference feature and the first time reference feature respectively, and determine the first fusion reference feature corresponding to the first spatial reference feature, the second fusion reference feature corresponding to the second spatial reference feature, and the third fusion reference feature corresponding to the first time reference feature, which are used as input data of the neural network model for learning to obtain the extended target video.
12. An electronic device, characterized in that: include: processor; A memory for storing a program for processing data generated by an electronic device, wherein when the program is read and executed by the processor, the video extension processing method as described in any one of claims 1 to 9 is executed.
13. A computer storage medium, characterized in that: It comprises a computer program, and when the computer program is run on an electronic device, the electronic device executes the video extension processing method as described in any one of claims 1 to 9.