Panoramic video generation method and device, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a panoramic video generation method and device, electronic equipment and a readable storage medium. The method comprises the steps of obtaining video frame data and a target text, and obtaining text features based on the target text; inputting the video frame data and the text features into a motion adapter, and performing image feature extraction on the video frame data to obtain image features; embedding the image feature and a time step corresponding to a preset time step for fusion to obtain a target fusion feature, and performing motion feature extraction on the target fusion feature and the text feature to obtain a motion feature; and performing diffusion processing based on the text features, the motion features and the noise map to obtain a panoramic video frame sequence, and obtaining a target panoramic video based on the panoramic video frame sequence. According to the method, the high-quality panoramic video frame sequence can be generated based on the motion features corresponding to the time steps, so that the high-quality target panoramic video is obtained.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a panoramic video generation method, device, electronic device and readable storage medium. Background Art

[0002] As an innovative way of visual experience, panoramic video is gradually becoming a popular choice for virtual reality and augmented reality applications because it can provide a perspective without blind spots. However, since panoramic videos need to capture scenes in a wider field of vision, the content distribution of panoramic videos is significantly different from that of standard videos. Secondly, the motion patterns in panoramic videos usually present curved trajectories, which is different from the straight-line motion in standard videos. This puts higher demands on motion prediction and compensation. Although existing text-to-video diffusion methods perform well in standard video generation, these existing methods often have poor results when generating high-quality panoramic videos due to the significant differences in content and motion patterns between panoramic videos and standard videos. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a panoramic video generation method, device, electronic device and readable storage medium to solve the problem in the prior art that high-quality panoramic videos cannot be generated.

[0004] In a first aspect of an embodiment of the present application, a method for generating a panoramic video is provided, the method comprising:

[0005] The video frame data and the target text corresponding to the video frame data are obtained, and text features are obtained based on the target text; the video frame data and the text features are input into a motion adapter, wherein the motion adapter includes an image feature processing module, a target fusion module and a motion feature processing module; the image feature processing module is used to extract image features from the video frame data to obtain image features corresponding to the video frame data; the target fusion module is used to fuse the image features with the time step embedding corresponding to the preset time step to obtain the target fusion feature, and the motion feature processing module is used to extract motion features from the target fusion feature and the text feature to obtain the motion feature; the text features, the motion features and the noise map are input into a diffusion model, and a plurality of U-net networks continuously set in the diffusion model are used to perform diffusion processing to obtain a panoramic video frame sequence output by the diffusion model, and a target panoramic video is obtained based on the panoramic video frame sequence.

[0006] According to a second aspect of the embodiments of the present application, a panoramic video generation device is provided, the method comprising:

[0007] The acquisition module is configured to acquire video frame data and target text corresponding to the video frame data, and obtain text features based on the target text; the input module is configured to input the video frame data and text features into a motion adapter, wherein the motion adapter includes an image feature processing module, a target fusion module and a motion feature processing module; the image feature processing module is configured to use the image feature processing module to extract image features from the video frame data to obtain image features corresponding to the video frame data; the motion feature processing module is configured to use the target fusion module to fuse the image features with the time step embedding corresponding to the preset time step to obtain target fusion features, and use the motion feature processing module to extract motion features from the target fusion features and text features to obtain motion features; the diffusion module is configured to input text features, motion features and noise maps into a diffusion model, use multiple U-net networks continuously set in the diffusion model to perform diffusion processing, obtain a panoramic video frame sequence output by the diffusion model, and obtain a target panoramic video based on the panoramic video frame sequence.

[0008] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0009] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0010] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0011] The video frame data and the target text corresponding to the video frame data are obtained, and text features are obtained based on the target text; the video frame data and the text features are input into a motion adapter, wherein the motion adapter includes an image feature processing module, a target fusion module and a motion feature processing module; the image feature processing module is used to extract image features from the video frame data to obtain image features corresponding to the video frame data; the target fusion module is used to fuse the image features with the time step embedding corresponding to the preset time step to obtain the target fusion feature, and the motion feature processing module is used to extract motion features from the target fusion feature and the text feature to obtain the motion feature; the text features, the motion features and the noise map are input into a diffusion model, and a plurality of U-net networks continuously set in the diffusion model are used to perform diffusion processing to obtain a panoramic video frame sequence output by the diffusion model, and a target panoramic video is obtained based on the panoramic video frame sequence. By extracting the image features of the video frame data and fusing them with the time step embedding of the preset time step, the model can capture different motion features in the generation stage of the panoramic video frame sequence to simulate the gradual refinement process when generating video content. At the same time, the process of fusing the image features with the time step embedding corresponding to the preset time step also makes it possible to consider the contextual information of the entire video in the generation stage of the panoramic video frame sequence, rather than focusing only on the information of a single frame. At the same time, the corresponding time step motion features obtained are combined with multiple U-net networks in the diffusion model for diffusion processing, so that a high-quality panoramic video frame sequence can be generated, thereby obtaining a high-quality target panoramic video. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 It is a flowchart of a panoramic video generation method provided in an embodiment of the present application;

[0014] Figure 2 It is a flowchart of another panoramic video generation method provided in an embodiment of the present application;

[0015] Figure 3 It is a schematic diagram of the working process of a motion adapter provided in an embodiment of the present application;

[0016] Figure 4 is a structural schematic diagram of a panoramic video generation device provided in an embodiment of the present application;

[0017] Figure 5It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0019] A panoramic video generation method, device, electronic device and readable storage medium according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0020] Figure 1 FIG. 1 is a flow chart of a panoramic video generation method provided in an embodiment of the present application. Figure 1 As shown, the method includes:

[0021] S101, obtaining video frame data and target text corresponding to the video frame data, and obtaining text features based on the target text.

[0022] Specifically, a video is composed of a series of continuous image frames, each frame is a static image of the video, and the video frame data can be obtained by extracting these static images from the video using video decoding and frame extraction. The target text refers to the text related to or describing the content of the video frame. Text features are an abstract representation of text information, which can help the terminal understand the meaning and context of the text. In order to extract features from the target text, various text processing techniques can be used, such as bag of words model, TF-IDF, word embedding, etc. (such as Word2Vec, BERT, etc.). Preferably, this embodiment uses CLIP text encoder to convert the target text into text features.

[0023] S102, inputting the video frame data and text features into a motion adapter, wherein the motion adapter includes an image feature processing module, a target fusion module and a motion feature processing module.

[0024] S103, extracting image features from the video frame data using an image feature processing module to obtain image features corresponding to the video frame data;

[0025] Specifically, the image feature processing module of the motion adapter is used to extract image features from the video frame data, thereby obtaining the image features corresponding to the video frame data. The motion adapter is a module for processing motion information in the video frame data. The image feature processing module refers to the network structure used to extract image features in the motion adapter module. When the video frame data is input into the motion adapter, the image feature processing module starts working. The module performs feature processing operations on the video frame layer by layer, thereby extracting image features from the video frame. These image features may include visual information such as color, texture, and edge, which are used for processing subsequent tasks.

[0026] S104, using a target fusion module to fuse the image feature with the time step embedding corresponding to the preset time step to obtain a target fusion feature, and using a motion feature processing module to extract motion features from the target fusion feature and the text feature to obtain a motion feature;

[0027] Specifically, time step embedding can encode temporal information into the model input. It is used to introduce time steps as additional features into the model and help the model learn time-related patterns and dynamics. Time step embedding is usually represented by a vector, where each time step has a corresponding time embedding vector. The purpose of time step embedding is to pass temporal information to the model together with other features so that it can capture changes and trends at different time points. Time step embedding corresponding to preset time steps. Characterization Before model training or application, the corresponding time embedding vector has been defined or calculated for each preset time step. When the model processes data at a specific time step, it can directly use these time embedding vectors as part of the input. The preset time step embedding helps the model process time-related data more efficiently and improves the performance of the model. The fusion of image features and time step embedding so that subsequent models can use both information at the same time, so that different motion features can be captured in the generation stage of the panoramic video frame sequence to simulate the gradual refinement process when generating video content. At the same time, the fusion of time steps allows the model to predict the main scene first and then refine the motion details during the generation process, simulating the process of human understanding and generation of motion videos.

[0028] As an example, the acquired target text contains a description of the main scene (such as "a person plays basketball on a basketball court") and a description of the details of the movement (such as "he dribbled the ball flexibly and suddenly jumped up to shoot"). By adopting the time step embedding fusion method, the model can first predict the main scene. That is, a basic basketball scene is generated based on the input text description, including key elements such as the basketball court and players. After predicting the main scene, the model will further refine the details of the movement. Through the fusion of time steps, the model can simulate the process of human understanding and generation of sports videos. When watching and describing sports videos, humans usually focus on the overall scene and main actions first, and then gradually notice the details and changes, that is, the model can decompose the video generation process into a series of time steps. In each time step, the model generates the content of the next frame based on the previous prediction results and the current text description.

[0029] S105, inputting text features, motion features and noise maps into a diffusion model, performing diffusion processing using a plurality of U-net networks continuously arranged in the diffusion model, obtaining a panoramic video frame sequence output by the diffusion model, and obtaining a target panoramic video based on the panoramic video frame sequence.

[0030] Specifically, when text features, motion features, and noise maps are input into the diffusion model, the model starts to work. The diffusion model is a deep generative model that generates new data by simulating the diffusion and denoising process of data. In this step, the model performs diffusion processing through multiple U-net networks set up in succession.

[0031] The U-net network is a neural network structure commonly used in image segmentation and generation tasks. Through the form of encoders and decoders, it captures multi-level information in the image and gradually restores the details of the image during the decoding process. In the diffusion model, multiple U-net networks are set up continuously, and each network processes the input data to a certain extent to simulate the diffusion process of the data. Through this series of diffusion processing, the model finally outputs a panoramic video frame sequence. This sequence contains multiple video frames generated based on the input features and noise maps. These frames are arranged in chronological order to form a target panoramic video. The target panoramic video contains both the content and style described by the input text and the dynamic changes of objects in the video, while maintaining the coherence and authenticity of the video.

[0032] According to the technical solution provided in the embodiment of the present application, video frame data and target text corresponding to the video frame data are obtained, and text features are obtained based on the target text; the video frame data and text features are input into a motion adapter, wherein the motion adapter includes an image feature processing module, a target fusion module and a motion feature processing module; the image feature processing module is used to extract image features from the video frame data to obtain image features corresponding to the video frame data; the target fusion module is used to fuse the image features with the time step embedding corresponding to the preset time step to obtain target fusion features, and the motion feature processing module is used to extract motion features from the target fusion features and text features to obtain motion features; the text features, motion features and noise maps are input into a diffusion model, and diffusion processing is performed using multiple U-net networks continuously arranged in the diffusion model to obtain a panoramic video frame sequence output by the diffusion model, and a target panoramic video is obtained based on the panoramic video frame sequence. By extracting the image features of the video frame data and fusing them with the time step embedding of the preset time step, different motion features can be captured in the generation stage of the panoramic video frame sequence to simulate the gradual refinement process when generating the video content. At the same time, the process of fusing the image features with the time step embedding corresponding to the preset time step also makes it possible to consider the contextual information of the entire video in the generation stage of the panoramic video frame sequence, rather than focusing only on the information of a single frame. Finally, the corresponding time step motion features are obtained and combined with multiple U-net networks in the diffusion model for diffusion processing to generate a high-quality panoramic video frame sequence, thereby obtaining a high-quality target panoramic video.

[0033] In some embodiments, a motion feature processing module is used to extract motion features from target fusion features and text features to obtain motion features, including:

[0034] The time step resampler with adaptive layer norm in the motion feature processing module is used to extract dynamic features from the target fusion features. The cross attention mechanism is used to perform multimodal fusion on the target fusion features and text features after dynamic feature extraction to obtain multimodal fusion features. The convolution network in the motion feature processing module is used to perform convolution processing on the multimodal fusion features to obtain motion features.

[0035] Specifically, the time step resampler of the adaptive layer norm is a technology for processing time series data. It can adaptively adjust the sensitivity to time information according to the dynamic characteristics of the data (such as the unevenness of the time step) to match the complexity of different video contents, so as to better capture the key information in the data.

[0036] In a further refined embodiment, the motion feature processing module implements dynamic feature extraction by the following steps: The target fusion feature is processed using a time step resampler with an adaptive layer norm. The purpose of this step is to adaptively resample the target fusion feature according to the dynamic characteristics of the target fusion feature to extract key information related to the motion. By adjusting the time step or sampling rate, the changes in motion features can be captured more effectively.

[0037] Furthermore, the target fusion features and text features after dynamic feature extraction are fused using the cross-attention mechanism. Information from different modalities is combined to fully utilize the complementarity between text features and dynamic features. By calculating the cross-attention score, the degree of association between different features can be determined and fused accordingly. Then, the multimodal fusion features are convolved using a convolutional network. The local and global information in the multimodal features is further extracted and integrated to generate a higher-level feature representation, namely the motion feature.

[0038] According to the technical solution provided in the embodiment of the present application, the time step resampler of the adaptive layer norm in the motion feature processing module is used to extract dynamic features from the target fusion features; the cross-attention mechanism is used to perform multimodal fusion on the target fusion features and text features after the dynamic feature extraction to obtain multimodal fusion features; the convolution network in the motion feature processing module is used to perform convolution processing on the multimodal fusion features to obtain motion features. By introducing the time step resampler of the adaptive layer norm and the cross-attention mechanism, the motion feature processing module can realize the effective extraction and fusion of the target fusion features and text features. Not only the dynamic characteristics of the data are considered, but also the complementarity between multimodal information is fully utilized, thereby improving the accuracy and reliability of motion feature extraction.

[0039] In some embodiments, the convolution network in the motion feature processing module is used to perform convolution processing on the multimodal fusion features to obtain motion features, including:

[0040] The multimodal fusion features are convolved using the spatial convolution layer of the convolutional network to obtain spatial convolution features. The spatial convolution features are corrected using the rectified linear unit, and the corrected spatial convolution features are convolved using the temporal convolution layer of the convolutional network to obtain spatiotemporal convolution features. The target fusion features are added to the spatiotemporal convolution features to obtain motion features.

[0041] Specifically, the spatial convolution layer in the convolutional network is responsible for convolution processing of the multimodal fusion features in the spatial dimension. By applying the spatial convolution layer, sliding on the input feature map and performing dot product operations, local spatial information can be extracted, which can capture the spatial structure in the multimodal fusion features and generate spatial convolution features.

[0042] It is understandable that in order to further enhance the expressiveness of features, the rectified linear unit (ReLU) can be used to perform nonlinear correction processing on spatial convolution features. The ReLU function can introduce nonlinear characteristics to help the network better fit complex data distribution. After being processed by the ReLU function, the negative values ​​in the feature are set to zero, while the positive values ​​remain unchanged, thereby retaining important feature information.

[0043] Furthermore, the temporal convolution layer in the convolutional network performs convolution processing on the modified spatial convolution features in the temporal dimension. The temporal convolution layer can capture the temporal dependency and dynamic changes in the feature sequence. By performing convolution operations on features of different time steps, the temporal information and dynamic patterns in the motion features can be extracted to generate spatiotemporal convolution features.

[0044] Furthermore, the target fusion feature is added to the spatiotemporal convolution feature to obtain the final motion feature. This step combines the target fusion feature with the spatiotemporal convolution feature after convolution, and comprehensively considers the information of the target fusion feature and the spatiotemporal convolution feature, so as to obtain a more complete and accurate motion feature representation, thereby improving the accuracy and coherence of the generated video.

[0045] According to the technical solution provided in the embodiment of the present application, the spatial convolution layer of the convolutional network is used to perform convolution processing on the multimodal fusion features in the spatial dimension to obtain spatial convolution features; the spatial convolution features are corrected using a rectified linear unit, and the corrected spatial convolution features are convoluted in the temporal dimension using a temporal convolution layer to obtain spatiotemporal convolution features; the target fusion features are added to the spatiotemporal convolution features to obtain motion features. By combining the convolution processing in the spatial and temporal dimensions, the local and global information in the multimodal fusion features can be fully extracted and integrated, and high-quality motion features can be generated to provide support for subsequent processing steps.

[0046] In some embodiments, the image feature processing module is used to extract image features from the video frame data to obtain image features corresponding to the video frame data, including:

[0047] The sampling network in the image feature processing module is used to downsample the video frame data; the convolutional neural network in the image feature processing module is used to perform convolution processing on the downsampled video frame data to obtain image features.

[0048] Specifically, as the first step of image feature extraction, the sampling network is responsible for downsampling the input video frame data. Downsampling can reduce the data size and computational complexity while retaining the key information of the original data. By selecting appropriate sampling strategies, such as average pooling, maximum pooling, etc., the sampling network can downsample the video frame data and obtain a smaller output size. This reduces the amount of computation for subsequent convolutional neural network processing and improves the efficiency of feature extraction.

[0049] Furthermore, the convolutional neural network performs convolution processing on the downsampled video frame data to extract image features. The convolutional neural network can learn local and global features in the image by stacking multiple convolutional layers, pooling layers, activation functions and other components. During the convolution process, the convolution kernel in the network slides on the input data and performs convolution operations to extract feature information of different scales and positions. After the superposition of multiple layers of convolution, the network can gradually capture higher-level abstract features and form a deep understanding of the content of the video frame. Finally, the output of the convolutional neural network is the image features corresponding to the video frame data.

[0050] According to the technical solution provided in the embodiment of the present application, the sampling network in the image feature processing module is used to downsample the video frame data; the convolutional neural network in the image feature processing module is used to perform convolution processing on the downsampled video frame data to obtain image features. By combining the processing capabilities of the sampling network and the convolutional neural network, the image feature processing module can realize efficient image feature extraction of video frame data.

[0051] In some embodiments, text features, motion features, and noise maps are input into a diffusion model, and diffusion processing is performed using a plurality of U-net networks continuously set up by the diffusion model to obtain a panoramic video frame sequence output by the diffusion model, including:

[0052] The text features are integrated into a plurality of U-net networks arranged in succession; the noise map is denoised using the first U-net network among the plurality of U-net networks to obtain a first denoising result output by the first U-net network; the motion features and the first denoising result are added to obtain a first target result; the first target result is input into a second U-net network among the plurality of U-net networks for denoising to obtain a second denoising result output by the second U-net network; the motion features and the second denoising result are added to obtain a second target result, and the denoising and addition processes of each U-net network are repeated until a target result output by the last U-net network among the plurality of U-net networks is obtained, wherein the target result represents a video frame sequence.

[0053] Specifically, the purpose of fusing text features into multiple U-net networks set up in succession is to incorporate the semantic information of the text description into the process of generating video frames. By combining text features with each level of the U-net network, the network can better understand and follow the guidance of the text description to generate video frame content that conforms to the text intent. Using the first U-net network in multiple U-net networks to denoise the noise map is the first step in generating video frames. As the initial input, the noise map contains randomly generated pixel values, representing the initial state of the video frame. Through the denoising process of the first U-net network, the network can gradually extract useful structural information from the noise and generate preliminary video frame content. The denoising process gradually optimizes the pixel values ​​to make the output image closer to the appearance of the target video frame.

[0054] Furthermore, the motion features of the corresponding time step are added to the first denoising result to obtain the first target result. This step integrates the motion features into the denoised image, and by combining the motion features containing spatial and temporal dimensions, the generated video frame has a dynamic and coherent motion effect. The motion features contain motion information and dynamic changes in the video frame sequence. By adding them to the denoising result, these motion characteristics can be integrated into the video frame to generate more realistic and natural video content.

[0055] Furthermore, the first target result is input into the second U-net network for denoising to obtain a second denoising result. This process is similar to the denoising process of the first U-net network, but the input is a video frame that has been integrated with motion features. Through further denoising and optimization of the second U-net network, the network can continue to improve the quality and details of the video frame, making it closer to the target video frame.

[0056] It can be understood that the denoising and addition processing of each U-net network is repeated, and the denoising results of each U-net network will be combined with the motion features of the corresponding time step as the input of the next U-net network until the target result output by the last U-net network is obtained. In this process, each U-net network further optimizes the quality and details of the video frame based on the previous network, and gradually improves the accuracy and authenticity of the generated video frame. The final output target result is a video frame sequence that conforms to the text description and motion features.

[0057] According to the technical solution provided in the embodiment of the present application, by integrating text features into multiple U-net networks arranged in succession, and using these networks to denoise the noise map and fuse motion features, a panoramic video frame sequence that conforms to the text description and has dynamic and coherent motion effects is generated.

[0058] In some embodiments, each U-net network in the plurality of U-net networks includes a picture layer;

[0059] The text features are integrated into a plurality of U-net networks arranged in a continuous manner, including: integrating the text features into the image layer of each U-net network in the plurality of U-net networks through a cross attention mechanism.

[0060] Specifically, in the U-net network structure, the picture layer is responsible for processing image data. In order to effectively integrate text features into the U-net network, this embodiment adopts a cross attention mechanism. This mechanism enables the U-net network to fully consider the semantic information of the text description when generating video frames.

[0061] According to the technical solution provided in this embodiment, the text features are fused into the image layer of each U-net network in multiple U-net networks through the cross attention mechanism. The text features are effectively fused into multiple U-net networks set up in succession, so that each U-net network can fully consider the guidance of the text description when processing image data. This improves the accuracy and authenticity of the generated video frames, making them more consistent with the semantic content of the text description.

[0062] In some embodiments, each U-net network in the plurality of U-net networks includes a motion layer, and before fusing the text features to the image layer of each U-net network in the plurality of U-net networks through a cross attention mechanism, further includes:

[0063] The parameters of the motion layer of each U-net network in the multiple U-net networks are fixed; the first frame of video is extracted from the video frame data, and the first frame of video is used to fine-tune the image layer in each U-net network to obtain a pre-trained image layer; each U-net network in the multiple U-net networks is constructed using the pre-trained image layer and motion layer.

[0064] Specifically, in order to improve the generation efficiency and stability of the model, this embodiment proposes a method for fixing the parameters of the motion layer in the U-net network. The motion layer is responsible for processing motion features, and fixing its parameters can avoid adjusting it during the training process, thereby reducing the amount of training calculations and time. At the same time, since motion features are usually relatively stable during the video frame generation process, fixing their parameters can also maintain the consistency of the model when processing motion features.

[0065] On the basis of fixing the motion layer parameters, this embodiment further proposes a method for fine-tuning the picture layer. First, the first frame of video is extracted from the video frame data. This is because the first frame of video usually contains the initial state and key information of the video sequence, which can be used as a reference for picture layer training. Then, the first frame of video is used to fine-tune the picture layer in each U-net network so that the picture layer can better understand and process the image data of the video frame. The pre-trained picture layer obtained after fine-tuning training will have stronger image processing capabilities and can better cooperate with the motion layer to generate high-quality video frames.

[0066] It can be understood that by using the pre-trained image layer and the fixed motion layer to build a complete U-net network, since the image layer has been fine-tuned and the motion layer parameters are fixed, such a U-net network will have higher generation efficiency and stability, and can generate video frame sequences that meet the text description and motion characteristics more quickly.

[0067] According to the technical solution provided in this embodiment, the motion layer parameters in the U-net network are fixed, and the picture layer is fine-tuned to improve the generation efficiency and stability of the model. At the same time, the U-net network is constructed by combining the pre-trained picture layer and the fixed motion layer, which can generate high-quality video frame sequences more quickly to meet the needs of panoramic video generation.

[0068] Figure 2 FIG. 1 is a flow chart of another panoramic video generation method provided in an embodiment of the present application. Figure 2 As shown:

[0069] The acquired target text is processed by a text encoder to obtain text features corresponding to the target text, and the video frame data and the text features are input into a motion encoder to obtain motion features processed by the motion encoder. The text features, motion features and noise maps are further input into a diffusion model, wherein the diffusion model includes a plurality of U-net networks arranged in succession, each U-net network includes a picture layer and a motion layer, and diffusion processing is performed using the plurality of U-net networks arranged in succession in the diffusion model to obtain a panoramic video frame sequence output by the diffusion model.

[0070] Figure 3 Schematic diagram of the working process of a motion adapter provided by an embodiment of the present application. Figure 3 As shown:

[0071] First, after the video frame data is input into the motion adapter, the sampling network in the image feature processing module of the motion adapter will downsample the video frame data, and the convolutional neural network will perform convolution processing on the downsampled video frame data to obtain image features.

[0072] Then, the target fusion module of the motion adapter fuses the image features with the time step embedding corresponding to the preset time step to obtain the target fusion features.

[0073] Then, the motion feature processing module of the motion adapter will use the time step resampler of the adaptive layer norm to extract dynamic features from the target fusion features. After the dynamic features are extracted, the target fusion features are multimodally fused with the text features input to the motion adapter using a cross-attention mechanism to obtain multimodal fusion features.

[0074] Finally, the convolution network in the motion feature processing module is used to perform convolution processing on the multimodal fusion features, wherein the convolution network includes a spatial convolution layer, a function layer and a temporal convolution layer. The convolution network performs convolution processing on the multimodal fusion features specifically including: performing convolution processing on the multimodal fusion features using the spatial convolution layer to obtain spatial convolution features; correcting the spatial convolution features using the rectified linear unit of the function layer, and convolving the corrected spatial convolution features using the temporal convolution layer to obtain spatiotemporal convolution features, and adding the target fusion features to the spatiotemporal convolution features to obtain motion features.

[0075] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the process of the embodiments of the present application.

[0076] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.

[0077] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0078] Figure 4 Schematic diagram of a panoramic video generation device provided in an embodiment of the present application. Figure 4 As shown, the device comprises:

[0079] The acquisition module 401 is configured to acquire the video frame data and the target text corresponding to the video frame data, and obtain text features based on the target text;

[0080] An input module 402 is configured to input the video frame data and text features into a motion adapter, wherein the motion adapter includes an image feature processing module, an object fusion module, and a motion feature processing module;

[0081] The image feature processing module 403 is configured to extract image features from the video frame data using the image feature processing module to obtain image features corresponding to the video frame data;

[0082] The motion feature processing module 404 is configured to use the target fusion module to fuse the image feature with the time step embedding corresponding to the preset time step to obtain the target fusion feature, and use the motion feature processing module to extract the motion feature of the target fusion feature and the text feature to obtain the motion feature;

[0083] The diffusion module 405 is configured to input text features, motion features and noise maps into the diffusion model, perform diffusion processing using multiple U-net networks continuously set in the diffusion model, obtain a panoramic video frame sequence output by the diffusion model, and obtain a target panoramic video based on the panoramic video frame sequence.

[0084] In some embodiments, the motion feature processing module 404 is also configured to use the time step resampler of the adaptive layer norm in the motion feature processing module to perform dynamic feature extraction on the target fusion feature; use the cross-attention mechanism to perform multimodal fusion on the target fusion feature and text feature after the dynamic feature extraction to obtain multimodal fusion features; use the convolution network in the motion feature processing module to perform convolution processing on the multimodal fusion features to obtain motion features.

[0085] In some embodiments, the motion feature processing module 404 is further configured to perform convolution processing on the multimodal fusion feature using the spatial convolution layer of the convolution network to obtain the spatial convolution feature; perform correction processing on the spatial convolution feature using the rectified linear unit, and perform convolution processing on the corrected spatial convolution feature using the temporal convolution layer of the convolution network to obtain the spatiotemporal convolution feature;

[0086] In some embodiments, the image feature processing module 403 is further configured to downsample the video frame data using the sampling network in the image feature processing module; and perform convolution processing on the downsampled video frame data using the convolutional neural network in the image feature processing module to obtain image features.

[0087] In some embodiments, the diffusion module 405 is also configured to fuse text features into multiple U-net networks arranged in succession; use the first U-net network among the multiple U-net networks to denoise the noise map to obtain a first denoising result output by the first U-net network; add the motion feature and the first denoising result to obtain a first target result; input the first target result into the second U-net network among the multiple U-net networks for denoising to obtain a second denoising result output by the second U-net network; add the motion feature and the second denoising result to obtain a second target result, and repeat the denoising and addition processing of each U-net network until the target result output by the last U-net network among the multiple U-net networks is obtained, and the target result represents the panoramic video frame sequence.

[0088] In some embodiments, each of the multiple U-net networks includes a picture layer, and the diffusion module 405 is further configured to fuse the text features into the picture layer of each of the multiple U-net networks through a cross-attention mechanism.

[0089] In some embodiments, each of the multiple U-net networks also includes a motion layer, and the diffusion module 405 is further configured to fix the parameters of the motion layer of each U-net network in the multiple U-net networks; extract a first frame of video from the video frame data, and use the first frame of video to fine-tune the image layer in each U-net network to obtain a pre-trained image layer; use the pre-trained image layer and motion layer to construct each U-net network in the multiple U-net networks.

[0090] Figure 5 Schematic diagram of an electronic device 5 provided in an embodiment of the present application. Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 501 executes the computer program 503, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0091] The electronic device 5 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 5 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5 The electronic device 5 is merely an example and does not limit the electronic device 5 , and may include more or less components than those shown in the figure, or different components.

[0092] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0093] The memory 502 may be an internal storage unit of the electronic device 5, for example, a hard disk or memory of the electronic device 5. The memory 502 may also be an external storage device of the electronic device 5, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. The memory 502 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 502 is used to store computer programs and other programs and data required by the electronic device.

[0094] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0095] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The readable storage medium may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0096] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A panoramic video generation method, characterized in that: include: Acquire video frame data and target text corresponding to the video frame data, and obtain text features based on the target text; Inputting the video frame data and the text features into a motion adapter, wherein the motion adapter comprises an image feature processing module, a target fusion module and a motion feature processing module; Using the image feature processing module to extract image features from the video frame data to obtain image features corresponding to the video frame data; The target fusion module is used to fuse the image feature with the time step embedding corresponding to the preset time step to obtain the target fusion feature, and the motion feature processing module is used to extract the motion feature of the target fusion feature and the text feature to obtain the motion feature; The text features, the motion features and the noise map are input into a diffusion model, and diffusion processing is performed using a plurality of U-net networks continuously arranged in the diffusion model to obtain a panoramic video frame sequence output by the diffusion model, and a target panoramic video is obtained based on the panoramic video frame sequence.

2. The method according to claim 1, characterized in that The step of using the motion feature processing module to extract motion features from the target fusion features and the text features to obtain motion features includes: Using the time step resampler of the adaptive layer norm in the motion feature processing module to extract dynamic features from the target fusion features; Using a cross attention mechanism, multimodal fusion is performed on the target fusion feature and the text feature after dynamic feature extraction to obtain a multimodal fusion feature; The multimodal fusion feature is convolved using the convolutional network in the motion feature processing module to obtain the motion feature.

3. The method according to claim 2, characterized in that The step of performing convolution processing on the multimodal fusion feature using the convolution network in the motion feature processing module to obtain the motion feature includes: Using the spatial convolution layer of the convolutional network to perform convolution processing on the multimodal fusion features to obtain spatial convolution features; The spatial convolution feature is corrected by using a rectified linear unit, and the corrected spatial convolution feature is convolved by using a temporal convolution layer of the convolution network to obtain a spatiotemporal convolution feature; The target fusion feature and the spatiotemporal convolution feature are added together to obtain the motion feature.

4. The method according to claim 1, characterized in that: The using the image feature processing module to extract image features from the video frame data to obtain image features corresponding to the video frame data includes: Downsampling the video frame data using a sampling network in the image feature processing module; The convolutional neural network in the image feature processing module is used to perform convolution processing on the downsampled video frame data to obtain the image features.

5. The method according to claim 1, characterized in that The text features, the motion features and the noise map are input into the diffusion model, and a plurality of U-net networks continuously set by the diffusion model are used for diffusion processing to obtain a panoramic video frame sequence output by the diffusion model, including: Fusion of the text features into the plurality of U-net networks arranged in series; Using a first U-net network among the multiple U-net networks to perform denoising on the noise graph, to obtain a first denoising result output by the first U-net network; Adding the motion feature and the first denoising result to obtain a first target result; Inputting the first target result into a second U-net network among the multiple U-net networks for denoising, and obtaining a second denoising result output by the second U-net network; The motion feature and the second denoising result are added to obtain a second target result, and the denoising process and the addition process of each U-net network are repeated until a target result output by the last U-net network among the multiple U-net networks is obtained, wherein the target result represents the panoramic video frame sequence.

6. The method according to claim 5, characterized in that Each of the plurality of U-net networks comprises a picture layer; The step of fusing the text features into the plurality of U-net networks arranged in succession comprises: The text features are fused into the image layer of each U-net network in the multiple U-net networks through a cross-attention mechanism.

7. The method according to claim 6, characterized in that Each of the multiple U-net networks further includes a motion layer, and before fusing the text features to the image layer of each of the multiple U-net networks through the cross attention mechanism, it also includes: Fixing the parameters of each U-net network motion layer in the plurality of U-net networks; Extracting a first frame of video from the video frame data, and using the first frame of video to fine-tune the image layer in each U-net network to obtain a pre-trained image layer; Each U-net network in the plurality of U-net networks is constructed using the pre-trained picture layer and the motion layer.

8. A panoramic video generation device, characterized in that: include: An acquisition module is configured to acquire video frame data and a target text corresponding to the video frame data, and obtain text features based on the target text; An input module configured to input the video frame data and the text features into a motion adapter, wherein the motion adapter includes an image feature processing module, a target fusion module, and a motion feature processing module; An image feature processing module is configured to extract image features from the video frame data using the image feature processing module to obtain image features corresponding to the video frame data; A motion feature processing module is configured to use the target fusion module to fuse the image feature with the time step embedding corresponding to the preset time step to obtain a target fusion feature, and use the motion feature processing module to extract motion features from the target fusion feature and the text feature to obtain a motion feature; The diffusion module is configured to input the text features, the motion features and the noise map into a diffusion model, perform diffusion processing using a plurality of U-net networks continuously arranged in the diffusion model, obtain a panoramic video frame sequence output by the diffusion model, and obtain a target panoramic video based on the panoramic video frame sequence.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images