Video space expansion method and device, equipment and storage medium

Abstract

The invention discloses a video space expansion method, device and equipment and a storage medium, and relates to the technical field of video generation, and the method comprises the steps: segmenting an original video through a video boundary time point predicted by a video boundary detection model, and obtaining a plurality of video segments; secondly, after down-sampling, performing video expansion by using a generative model to obtain a low-resolution externally-expanded video, extracting an externally-expanded part, and performing super-division processing on the externally-expanded part through a video super-division model to obtain a high-resolution video of the externally-expanded part; and calculating and matching gradient information between the video clip and the high-resolution video of the external expansion part by using a video fusion model, and expanding the video clip into the high-resolution video of the external expansion part to obtain a fused high-resolution external expansion video. The original video is seamlessly replaced into the extended video through the video fusion method, high-quality spatial extension is ensured, the content of the original video is kept, the creativity of the video generation model is utilized, and resource consumption is greatly reduced.

Description

Technical Field

[0001] The present application relates to the field of video generation technology, and in particular to a video space expansion method, device, equipment and storage medium. Background Art

[0002] Video Outpainting is a technique based on deep learning and computer vision that aims to generate additional areas from existing video content to expand it spatially, thereby obtaining a wider viewing angle or filling in missing parts of the video frame. This technique is similar to Image Outpainting and is often used to enhance the richness of video content or adapt to different display devices. Video Outpainting is more complex than Image Outpainting because the video has continuous frames and the expanded video needs to be continuous and consistent in time.

[0003] Current video spatial expansion usually relies on manual editing or rule-based methods, such as interpolation techniques, which linearly interpolate missing or damaged video frames or temporally interpolate based on neighboring frames, or texture synthesis-based completion, which manually or semi-automatically fills in missing areas by copying textures from existing images. Although these methods can handle static scenes, they often have poor results for dynamic scenes, complex textures, and object motion, and cannot handle videos with rich details.

[0004] Video generation technology based on denoising models has made significant progress in recent years, and can generate high-quality videos using text or image inputs. However, when these technologies are applied to the spatial expansion of videos, they face some challenges, such as high computing resource requirements, high processing complexity, and difficulty in processing dynamic content. Therefore, the current video spatial expansion methods based on video generation models are less effective when processing dynamic videos and consume too much resources. However, there are still many technical challenges to achieve high-quality, low-resource video spatial expansion. Summary of the invention

[0005] The purpose of this application is to overcome the existing technical defects and provide a video space expansion method, device, equipment and storage medium, which adopts a video generation model and a strategy of expansion through video downsampling to solve the problem that the current video space expansion method has poor effect in dynamic video and consumes too much resources.

[0006] The purpose of this application is achieved through the following technical solutions: In a first aspect, the present application proposes a video space expansion method, the method comprising: The original video is segmented into multiple video clips according to the video boundary time points predicted by the video boundary detection model; After downsampling the video clips, the generative model is used to perform video expansion to obtain a low-resolution expanded video; Extract the outward expansion part from the low-resolution outward expansion video and perform super-resolution processing on the outward expansion part through a video super-resolution model to obtain a high-resolution video of the outward expansion part; The video fusion model is used to calculate and match the gradient information between the video clip and the high-resolution video of the expanded part, and the video clip is expanded to the high-resolution video of the expanded part based on the gradient information to obtain a fused high-resolution expanded video.

[0007] In a possible implementation, the video boundary detection model includes a video feature extraction backbone, a feature similarity calculation module, and a plurality of classification heads; The video feature extraction backbone includes 3D convolution modules and Transformers modules, which are used to calculate the features of video frames; The feature similarity calculation module calculates the feature similarity vectors of adjacent frames and the RGB histogram similarity feature vectors of adjacent frames; Each classification head consists of multiple fully connected layers to predict the probability that a video frame is a video boundary.

[0008] In a possible implementation, the generative model includes a video description model and a video expansion model. After downsampling the video clip, the step of using the generative model to expand the video to obtain a low-resolution expanded video includes: Downsampling the video clip to obtain a low-resolution original video clip; Use the video description model to obtain corresponding text descriptions for low-resolution original video clips; The video expansion model is used to combine the low-resolution original video clips and the corresponding text descriptions to obtain a low-resolution expanded video.

[0009] In a possible implementation, the video expansion model includes a text feature extractor, a video feature extractor, a video encoder, a video decoder, and a video denoising model. The step of using the video expansion model in combination with the text description and the low-resolution original video clip to perform video expansion to obtain a low-resolution expanded video includes: Use a text feature extractor to extract features from the text description to obtain text features; Use a video feature extractor to process low-resolution original video clips frame by frame to obtain video features; Initialize the video expansion mask with the area pixel value of the low-resolution original video clip as 1 and the pixel value of the expanded area as 0; Based on the video expansion mask, the low-resolution original video segment is padded according to the target size to be expanded, and then the padded video segment is processed by a video encoder to obtain the first latent space feature; The second latent space feature is obtained by randomly initializing the target size to be expanded through Gaussian noise; Inputting text features, video features, video expansion mask, first latent space features, and second latent space features into a video denoising model to predict noise; A forward sampler is used to subtract noise from the second latent space feature to obtain a third latent space feature, and the third latent space feature, text feature, video feature, video expansion mask, and first latent space feature are input into the video denoising model again to perform noise prediction loop iteration to obtain a denoising result; The denoising result is processed using a video decoder to obtain a low-resolution out-scaled video.

[0010] In a possible implementation, the video denoising model adopts a U-shaped network structure, including a three-dimensional convolution module, a spatial attention module, a text attention module, and a layout attention module.

[0011] In a possible implementation, the step of using a video fusion model to calculate and match gradient information between a video clip and a high-resolution video of an expanded portion, and expanding the video clip into the high-resolution video of the expanded portion based on the gradient information to obtain a fused high-resolution expanded video includes: The gradient field of the video clip is calculated frame by frame through the Sobel operator, and the gradient divergence of the video clip is calculated using the Laplace operator; Based on the gradient field and gradient divergence, the edge pixel values ​​of the high-resolution video of the expanded part are calculated by using discrete solution equations; Inserting edge pixel values ​​into the high-resolution video of the expanded part obtains a fused high-resolution expanded video.

[0012] In a possible implementation, the expression of the discrete solution equation is: + + ,in, is the pixel horizontal coordinate, is the pixel ordinate, To solve the coordinates The pixel value at for The pixel value at for The pixel value at for The pixel value at for The pixel value at For video clips The gradient at .

[0013] In a second aspect, the present application proposes a video space expansion device, the device comprising: A video splitting module is used to split the original video into multiple video segments according to the video boundary time points predicted by the video boundary detection model; A video expansion module is used to downsample the video clips and then expand the video using a generative model to obtain a low-resolution expanded video; The video super-resolution module is used to extract the outward expansion part from the low-resolution outward expansion video and perform super-resolution processing on the outward expansion part through the video super-resolution model to obtain a high-resolution video of the outward expansion part; The video fusion module is used to calculate and match the gradient information between the video clip and the high-resolution video of the expanded part using the video fusion model, and expand the video clip to the high-resolution video of the expanded part based on the gradient information to obtain a fused high-resolution expanded video.

[0014] In a third aspect, the present application further proposes a computer device, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the video space expansion method as described in any one of the first aspects.

[0015] In a fourth aspect, the present application further proposes a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the video space expansion method as described in any one of the first aspects.

[0016] The above main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application; and in the present application, (non-conflicting options) options and other options can also be freely combined. After understanding the scheme of the present application, those skilled in the art can understand that there are many combinations based on the existing technology and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0017] The present application discloses a method, device, equipment and storage medium for video space expansion, which relates to the field of video generation technology. First, the original video is segmented into multiple video clips by the video boundary time points predicted by the video boundary detection model. Secondly, the video is expanded outward using the generative model after downsampling to obtain a low-resolution expanded video. The expanded part is extracted and the expanded part is super-resolved by the video super-resolution model to obtain a high-resolution video of the expanded part. The gradient information between the video clip and the high-resolution video of the expanded part is calculated and matched by the video fusion model, and the video clip is expanded to the high-resolution video of the expanded part to obtain the fused high-resolution expanded video. The original video is seamlessly replaced with the expanded video by the video fusion method, which ensures high-quality spatial expansion, maintains the content of the original video, utilizes the creativity of the video generation model and greatly reduces resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic flow chart of a video space expansion method proposed in an embodiment of the present application is shown.

[0020] Figure 2 A schematic diagram showing an example result of video super-resolution generation proposed in an embodiment of the present application is shown.

[0021] Figure 3 A schematic diagram showing an example result of first frame image generation proposed in an embodiment of the present application is shown.

[0022] Figure 4 A schematic diagram of an example result of video fusion generation proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0024] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of this application.

[0025] In the existing technology, video generation technology based on denoising models has made significant progress in recent years, and can generate high-quality videos using text or picture inputs. However, when these technologies are applied to the spatial expansion of videos, they face some challenges, such as high computing resource requirements, high processing complexity, and difficulty in processing dynamic content. Therefore, the current video spatial expansion methods based on video generation models are less effective when processing dynamic videos and consume too much resources. However, there are still many technical challenges to achieve high-quality, low-resource-consumption video spatial expansion.

[0026] Therefore, in order to solve the above technical problems, the embodiments of the present application propose a method, device, equipment and storage medium for video space expansion. The scheme uses a video shot segmentation method to divide the video into video clips of multiple different scenes, downsamples the video clips and uses a generative video expansion method to achieve spatial expansion of low-resolution videos, and then uses a video super-resolution algorithm to improve the resolution of the expanded part of the video. Finally, a video fusion algorithm is used to replace the original video with the expanded video to obtain a high-quality video of the target size. A high-quality expanded video is obtained by low-resolution video expansion and video super-resolution, and the original video is seamlessly replaced with the expanded video by a video fusion method, which not only ensures high-quality spatial expansion but also maintains the content of the original video, but also makes full use of the creativity of the video generation model and greatly reduces resource consumption. It is described in detail below.

[0027] Please refer to Figure 1 , Figure 1 A schematic diagram of a process of a video space expansion method proposed in an embodiment of the present application is shown, and the method includes: Step S1, dividing the original video into multiple video segments according to the video boundary time points predicted by the video boundary detection model.

[0028] The video boundary detection model includes a video feature extraction backbone, a feature similarity calculation module, and multiple classification heads; The video feature extraction backbone includes 3D convolution modules and Transformers modules, which are used to calculate the features of video frames; The feature similarity calculation module calculates the feature similarity vectors of adjacent frames and the RGB histogram similarity feature vectors of adjacent frames; Each classification head consists of multiple fully connected layers to predict the probability that a video frame is a video boundary.

[0029] In step S1, the feature information of each frame is first extracted from the original video using a 3D convolution module plus a Transformers module. 3D convolution can capture changes in the time dimension (the relationship between consecutive frames), and the Transformers module is used to understand the global context information. After that, for two adjacent frames, the similarity between them based on the extracted feature vectors is calculated; at the same time, the difference in the RGB color histogram between adjacent frames is considered as a supplementary feature. Then, a classification head composed of multiple fully connected layers is used to analyze each pair of adjacent frames and output the probability score of the existence of a boundary at that position. If a position scores high, it is considered that this is likely to be a scene transition point or boundary. Finally, the true boundary time point is determined based on the probability score obtained in the previous step, and the entire video is divided into several independent scene segments accordingly.

[0030] Step S2: After downsampling the video clip, the generative model is used to perform video expansion to obtain a low-resolution expanded video.

[0031] In order to reduce the amount of video data and reduce the computational complexity of subsequent processing, the original video frames are downsampled to reduce the size of each frame image, and a smooth transition is achieved by simply discarding some pixels or applying some form of filter (such as bilinear interpolation, Gaussian filtering, etc.).

[0032] In order to restore a high-resolution version from a low-resolution video and improve the spatial detail expression of the video, a generative model (deep learning model, such as convolutional neural network CNN or more advanced architectures such as GANs) is used to process the downsampled video clips. The model learns the mapping relationship between a large number of high-low resolution image pairs to predict and generate missing high-frequency information while maintaining visual quality.

[0033] Step S2 includes: Step S21, downsampling the video clip to obtain a low-resolution original video clip; Step S22: using the video description model to obtain a corresponding text description for the low-resolution original video clip; Step S23: Use the video expansion model in combination with the low-resolution original video clip and the corresponding text description to perform video expansion to obtain a low-resolution expanded video.

[0034] The generative model includes a video description model and a video expansion model. In order to reduce the resolution of the video and reduce the amount of data, video clip downsampling is used. Downsampling operations are performed on the original video clips, and the size of each frame image is reduced through methods such as average pooling and bilinear interpolation. For example, a high-definition video is converted into a standard-definition or lower-resolution version.

[0035] The video description model is then used to process the low-resolution original video clips and generate text descriptions corresponding to each video clip, which contain the main content or key information of the scene. The pre-trained video description model is used to input a low-resolution video clip, and the output is a natural language description of the video clip.

[0036] Combine text descriptions with low-resolution video clips to generate out-of-bound videos, and use additional contextual information (text descriptions) to improve the quality of video out-of-bounds. Take low-resolution video clips and corresponding text descriptions as input, and use a deep learning model specially designed for video out-of-bounds as the video out-of-bounds model. The model must not only be able to map images from low resolution to high resolution, but also be able to understand and integrate the given text information. The final output is a new version of the video with a higher resolution but still retaining the main content and style of the original video.

[0037] Step S23 includes: Use a text feature extractor to extract features from the text description to obtain text features; Use a video feature extractor to process low-resolution original video clips frame by frame to obtain video features; Initialize the video expansion mask with the area pixel value of the low-resolution original video clip as 1 and the pixel value of the expanded area as 0; Based on the video expansion mask, the low-resolution original video segment is padded according to the target size to be expanded, and then the padded video segment is processed by a video encoder to obtain the first latent space feature; The second latent space feature is obtained by randomly initializing the target size to be expanded through Gaussian noise; Inputting text features, video features, video expansion mask, first latent space features, and second latent space features into a video denoising model to predict noise; A forward sampler is used to subtract noise from the second latent space feature to obtain a third latent space feature, and the third latent space feature, text feature, video feature, video expansion mask, and first latent space feature are input into the video denoising model again to perform noise prediction loop iteration to obtain a denoising result; The denoising result is processed using a video decoder to obtain a low-resolution out-scaled video.

[0038] The video expansion model includes a text feature extractor, a video feature extractor, a video encoder, a video decoder and a video denoising model. The video denoising model adopts a U-shaped network structure, including a three-dimensional convolution module, a spatial attention module, a text attention module and a layout attention module.

[0039] Figure 3The example result schematic diagram of the first frame image generation proposed in the embodiment of the present application is shown. A text feature extractor is used to analyze the provided text description and extract relevant feature vectors that can represent the meaning or context of the text. Then, the input low-resolution original video clip is processed frame by frame using a video feature extractor to identify and extract important visual elements and structural features in each frame.

[0040] Create a mask (video expansion mask) for the original video, in which the pixel values ​​in the original video area are set to 1, and the pixel values ​​in the new area to be expanded are set to 0. According to the target size requirement, the original video is padded to the corresponding size and converted into the first latent space feature through the video encoder. In addition, based on the same final size specification, another latent representation is randomly generated using Gaussian distribution as the initial guess (second latent space feature).

[0041] All prepared data (text features, video features, video expansion mask, first latent space features, and second latent space features) are input together into a model designed specifically for predicting noise. The predicted noise part is subtracted from the second latent space features using the forward sampling method to obtain a new potential representation (third latent space feature). The updated third latent space features are re-input into the denoising model together with other unchanged information, and the noise prediction and removal operations are repeated until the ideal denoising effect is achieved. Finally, the data after multiple rounds of optimization processing is converted back to a normal video format through a video decoder to obtain an enhanced video with a higher resolution than the original.

[0042] Step S3: extracting the outward expansion part from the low-resolution outward expansion video and performing super-resolution processing on the outward expansion part through a video super-resolution model to obtain a high-resolution video of the outward expansion part.

[0043] Use the created video expansion mask to extract the expansion area from the low-resolution expansion video. This mask marks the original video area as 1 and the expansion area as 0, so the mask can be used to accurately locate and extract the expansion part, and then input the extracted expansion part into a pre-trained video super-resolution model. The task of the video super-resolution model is to predict and generate high-resolution images by learning the mapping relationship between low-resolution images and high-resolution images. The model processes the expansion part frame by frame, and each frame is processed by the model to generate a high-resolution version. Finally, these high-resolution frames are combined to form a high-resolution video of the expansion part.

[0044] Step S4: use the video fusion model to calculate and match the gradient information between the video clip and the high-resolution video of the expanded part, and based on the gradient information, expand the video clip to the high-resolution video of the expanded part to obtain a fused high-resolution expanded video.

[0045] Figure 4 A schematic diagram of an example result of video fusion generation proposed in an embodiment of the present application is shown. A video fusion model is used to calculate the gradient information between the original video clip and the high-resolution video of the expanded part, which reflects the rate of change of the image or video frame in space, and is particularly important for details such as edges and textures. Then, by matching the gradient information of the two parts, the visual consistency and continuity of the original video and the expanded part can be ensured. Based on the calculated gradient information, the video fusion model adjusts and merges the original video clip and the high-resolution video of the expanded part. In order to achieve a natural fusion effect, other factors such as color consistency, lighting conditions, etc. are considered to further optimize the final result. The original video content is seamlessly embedded in the high-resolution video of the expanded part to form a complete, high-quality target size video.

[0046] Step S4 includes: The gradient field of the video clip is calculated frame by frame through the Sobel operator, and the gradient divergence of the video clip is calculated using the Laplace operator; Based on the gradient field and gradient divergence, the edge pixel values ​​of the high-resolution video of the expanded part are calculated by using discrete solution equations; Inserting edge pixel values ​​into the high-resolution video of the expanded part obtains a fused high-resolution expanded video.

[0047] The original low-resolution video clip is used as the source video, and the high-resolution video of the expanded part after super-resolution processing is used as the target video. By applying the Sobel operator frame by frame, the gradient field of each frame can be obtained. By applying the Laplace operator to the gradient field, the gradient divergence can be calculated, which represents the rate of change of the gradient. The edge pixel value of the corresponding target video frame is calculated using the discrete solution equation. The discrete solution equation maps the gradient information of the source video to the target video to ensure the visual consistency of the two. These edge pixel values ​​are inserted into the corresponding positions of the target video, thereby achieving seamless fusion of the video clip and the expanded part.

[0048] The expression of the discrete solution equation is: + + ,in, is the pixel horizontal coordinate, is the pixel ordinate, To solve the coordinates The pixel value at for The pixel value at for The pixel value at for The pixel value at for The pixel value at For video clips The gradient at .

[0049] In the discrete solution equation, the pixel value (up, down, left, right) and the gradient value of the current position in the source video are used to update the pixel value at the current position. In this way, it can be ensured that the pixel value in the target video is smooth in the local area and can match the gradient information in the source video.

[0050] Compared with the prior art, the embodiments of the present application have the following beneficial effects: First, by performing shot boundary detection and segmentation on the original video through video splitting, different scenes or shots can be effectively identified.

[0051] Second, video expansion not only increases the spatial size of the video, but also retains key visual information, while also reducing the demand for hardware resources because the main operations are performed at a lower resolution.

[0052] Third, the previously obtained low-resolution extension part is further enhanced to a high-resolution state through video super-resolution, thereby ensuring that the entire output video has a uniform and high-quality level.

[0053] Fourth, a smooth transition from the original video to the extended version is achieved, creating a natural-looking and coherent picture even in the newly added areas that did not originally exist in the original video.

[0054] A possible implementation of a video space expansion device is provided below, which is used to execute each execution step and corresponding technical effect of the video space expansion method shown in the above embodiment and possible implementation. The device includes: A video splitting module is used to split the original video into multiple video segments according to the video boundary time points predicted by the video boundary detection model; A video expansion module is used to downsample the video clips and then expand the video using a generative model to obtain a low-resolution expanded video; The video super-resolution module is used to extract the outward expansion part from the low-resolution outward expansion video and perform super-resolution processing on the outward expansion part through the video super-resolution model to obtain a high-resolution video of the outward expansion part. Figure 2 A schematic diagram showing an example result of video super-resolution generation proposed in an embodiment of the present application is shown.

[0055] The video fusion module is used to calculate and match the gradient information between the video clip and the high-resolution video of the expanded part using the video fusion model, and expand the video clip to the high-resolution video of the expanded part based on the gradient information to obtain a fused high-resolution expanded video.

[0056] This preferred embodiment provides a computer device, which can implement the steps in any embodiment of the video space expansion method provided in the embodiments of the present application. Therefore, the beneficial effects of the video space expansion method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0057] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present application provides a storage medium, which stores a plurality of instructions, which can be loaded by a processor to execute the steps of any embodiment of the video space expansion method provided in the embodiment of the present application.

[0058] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0059] Since the instructions stored in the storage medium can execute the steps in any video space expansion method embodiment provided in the embodiments of the present application, the beneficial effects that can be achieved by any video space expansion method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0060] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A video space expansion method, characterized in that: The method comprises: The original video is segmented into multiple video clips according to the video boundary time points predicted by the video boundary detection model; After downsampling the video clips, the generative model is used to perform video expansion to obtain a low-resolution expanded video; Extract the outward expansion part from the low-resolution outward expansion video and perform super-resolution processing on the outward expansion part through a video super-resolution model to obtain a high-resolution video of the outward expansion part; The video fusion model is used to calculate and match the gradient information between the video clip and the high-resolution video of the expanded part, and the video clip is expanded to the high-resolution video of the expanded part based on the gradient information to obtain a fused high-resolution expanded video.

2. The video space expansion method according to claim 1, characterized in that: The video boundary detection model includes a video feature extraction backbone, a feature similarity calculation module, and multiple classification heads; The video feature extraction backbone includes 3D convolution modules and Transformers modules, which are used to calculate the features of video frames; The feature similarity calculation module calculates the feature similarity vectors of adjacent frames and the RGB histogram similarity feature vectors of adjacent frames; Each classification head consists of multiple fully connected layers to predict the probability that a video frame is a video boundary.

3. The video space expansion method according to claim 1, characterized in that: The generative model includes a video description model and a video expansion model. After downsampling the video clip, the generative model is used to expand the video to obtain a low-resolution expanded video, including: Downsampling the video clip to obtain a low-resolution original video clip; Use the video description model to obtain corresponding text descriptions for low-resolution original video clips; The video expansion model is used to combine the low-resolution original video clips and the corresponding text descriptions to obtain a low-resolution expanded video.

4. The video space expansion method according to claim 3, characterized in that: The video expansion model includes a text feature extractor, a video feature extractor, a video encoder, a video decoder, and a video denoising model. The steps of expanding the video by combining the video expansion model with the text description and the low-resolution original video clip to obtain a low-resolution expanded video include: Use a text feature extractor to extract features from the text description to obtain text features; Use a video feature extractor to process low-resolution original video clips frame by frame to obtain video features; Initialize the video expansion mask with the area pixel value of the low-resolution original video clip as 1 and the pixel value of the expanded area as 0; Based on the video expansion mask, the low-resolution original video segment is padded according to the target size to be expanded, and then the padded video segment is processed by a video encoder to obtain the first latent space feature; The second latent space feature is obtained by randomly initializing the target size to be expanded through Gaussian noise; Inputting text features, video features, video expansion mask, first latent space features, and second latent space features into a video denoising model to predict noise; A forward sampler is used to subtract noise from the second latent space feature to obtain a third latent space feature, and the third latent space feature, text feature, video feature, video expansion mask, and first latent space feature are input into the video denoising model again to perform noise prediction loop iteration to obtain a denoising result; The denoising result is processed using a video decoder to obtain a low-resolution out-scaled video.

5. The video space expansion method according to claim 4, characterized in that: The video denoising model adopts a U-shaped network structure, including a three-dimensional convolution module, a spatial attention module, a text attention module, and a layout attention module.

6. The video space expansion method according to claim 1, characterized in that: The steps of using a video fusion model to calculate and match gradient information between the video clip and the high-resolution video of the expanded part, and expanding the video clip to the high-resolution video of the expanded part based on the gradient information to obtain a fused high-resolution expanded video include: The gradient field of the video clip is calculated frame by frame through the Sobel operator, and the gradient divergence of the video clip is calculated using the Laplace operator; Based on the gradient field and gradient divergence, the edge pixel values ​​of the high-resolution video of the expanded part are calculated by using discrete solution equations; Inserting edge pixel values ​​into the high-resolution video of the expanded part obtains a fused high-resolution expanded video.

7. The video space expansion method according to claim 6, characterized in that: The expression of the discrete solution equation is: + + ,in, is the pixel horizontal coordinate, is the pixel ordinate, To solve the coordinates The pixel value at for The pixel value at for The pixel value at for The pixel value at for The pixel value at For video clips The gradient at .

8. A video space expansion device, characterized in that: The device comprises: A video splitting module is used to split the original video into multiple video segments according to the video boundary time points predicted by the video boundary detection model; A video expansion module is used to downsample the video clips and then expand the video using a generative model to obtain a low-resolution expanded video; The video super-resolution module is used to extract the outward expansion part from the low-resolution outward expansion video and perform super-resolution processing on the outward expansion part through the video super-resolution model to obtain a high-resolution video of the outward expansion part; The video fusion module is used to calculate and match the gradient information between the video clip and the high-resolution video of the expanded part using the video fusion model, and expand the video clip to the high-resolution video of the expanded part based on the gradient information to obtain a fused high-resolution expanded video.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the video space expansion method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is loaded and executed by a processor to implement the video space expansion method according to any one of claims 1 to 7.

Images