Video generation model training method, 3D scene generation method and device

Abstract

The invention relates to the technical field of virtual reality, and provides a video generation model training method, a 3D scene generation method and a 3D scene generation device, and the training method comprises the steps: obtaining a sample 360-degree video frame sequence and a corresponding sample description text; extracting a sample video frame feature sequence and a corresponding sample text feature, and adding a random noise tag to the sample video frame feature sequence; inputting the noise-added sample video frame feature sequence into a main network; determining a plurality of video frames with non-overlapped visual angles in the sample video frame feature sequence as key frames, and performing cross attention processing on each key frame; inputting the sample video frame feature sequence and the sample text features into a time sequence attention module of the main network to train parameters of LoRA so as to output predicted noise; and substituting predicted noise and random noise labels into the loss function, and completing model training when the loss function converges. The 360-degree video frame sequence output by the trained video generation model has good continuity, and a good 3D scene can be generated.

Description

Technical Field

[0001] This invention relates to the field of virtual reality technology, and in particular to a video generation model training method, a 3D scene generation method, and an apparatus. Background Technology

[0002] With the development of virtual reality and augmented reality technologies, the demand for 360-degree 3D scene generation is increasing across various fields. However, most existing 3D scene generation technologies are based on image generation methods, which suffer from problems such as geometric distortion and structural inconsistencies. Traditional generative models such as Stable Diffusion, while performing excellently in image generation, often face challenges in handling wide fields of view and complex geometric structures when dealing with 360-degree scenes. The generated 360-degree 3D scenes often involve a trade-off between global structure and local detail, leading to structural inconsistencies, local geometric distortion, and loss of detail in the images. Summary of the Invention

[0003] This invention provides a video generation model training method, a 3D scene generation method and apparatus to solve the problem that existing technologies generate 360-degree 3D scenes that have a trade-off between global structure and local details, resulting in inconsistent image structure, local geometric distortion and lack of detail.

[0004] This invention provides a method for training a video generation model, the video generation model comprising: an image encoder, a text encoder, and a main network, the training method comprising the following steps: Obtain the 360° video frame sequence of the sample and the corresponding sample description text; The sample 360° video frame sequence and the corresponding sample description are input into the image encoder and the text encoder respectively to obtain the sample video frame feature sequence and the corresponding sample text feature output by the image encoder and the text encoder respectively, and random noise labels are added to the sample video frame feature sequence. The noisy sample video frame feature sequence is input into the residual network module and self-attention module of the main network for processing; Multiple video frames with non-overlapping viewpoints in the sample video frame feature sequence processed by the residual network module and the self-attention module are identified as key frames, and cross-attention processing is performed on each key frame. The sample text features and the sample video frame feature sequence after cross-attention processing are input into the temporal attention module of the main network to train the parameters of LoRA, so as to output the predicted noise; Substitute the predicted noise and random noise labels into the loss function, and the model training is complete when the loss function converges. The trained video generation model is used to output a 360° video frame sequence of the target based on the input target random noise and the descriptive text corresponding to the 3D scene to be generated.

[0005] According to a video generation model training method provided by the present invention, the step of inputting the sample text features and the sample video frame feature sequence after cross-attention processing into the temporal attention module of the main network to train the parameters of LoRA to output predicted noise includes: The temporal attention module uses n video frame features as a sliding window to process the sample video frame feature sequence after cross-attention processing. The sliding window processing involves performing temporal attention processing on the sample video frame features and the sample text features in the sliding window, outputting prediction noise based on the sample video frame feature sequence after temporal attention processing, and training the parameters of LoRA. In the sample video frame feature sequence after cross-attention processing, the n video frame features in the previous window and the n video frame features in the next window have m overlapping video frame features, where m is less than n and n is less than the total number of video frames in the sample 360° video frame sequence. The average feature is calculated for the m overlapping video frame features and used to replace the m overlapping video frame features.

[0006] According to the video generation model training method provided by the present invention, the loss function is a stablediffusion paradigm loss function.

[0007] According to the present invention, a video generation model training method is provided, which obtains a 360° video frame sequence of samples and corresponding sample description text, including: The 360° video frame sequence of the sample is obtained by projecting the panoramic image. Input the 360° video frame sequence of the sample into the text annotation model to generate sub-description text corresponding to each video frame; Multiple sub-description texts are input into a large model to obtain the sample description text output by the large model.

[0008] The present invention also provides a 3D scene generation method, comprising the following steps: Input the target random noise and the target description text corresponding to the 3D scene to be generated into the video generation model to obtain the target 360° video frame sequence output by the video generation model; The target 360° video frame sequence is reconstructed to generate a 3D scene; The video generation model is trained based on any of the video generation model training methods described above.

[0009] According to a 3D scene generation method provided by the present invention, the video generation model further includes: an image decoder; the step of inputting target random noise and target description text corresponding to the 3D scene to be generated into the video generation model to obtain a target 360° video frame sequence output by the video generation model includes: The target description text is input into the text encoder to obtain the target text features output by the text encoder; The target random noise and the target text features are input into the main network of the video generation model to obtain the target video frame feature sequence output by the main network. The target video frame feature sequence is input into the image decoder to obtain the target 360° video frame sequence output by the image decoder.

[0010] The present invention also provides a 3D scene generation device, comprising the following modules: The model calling module is used to input the target random noise and the target description text corresponding to the 3D scene to be generated into the video generation model to obtain the target 360° video frame sequence output by the video generation model. The 3D scene reconstruction module is used to reconstruct the target 360° video frame sequence to generate a 3D scene; The video generation model is trained based on any of the video generation model training methods described above.

[0011] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the video generation model training method as described above, or to implement the 3D scene generation method as described above.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video generation model training method as described above, or implements the 3D scene generation method as described above.

[0013] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the video generation model training method as described above, or implements the 3D scene generation method as described above.

[0014] The video generation model training method, 3D scene generation method, and apparatus provided by this invention establish the correlation between noise, descriptive text, and 360° video frame sequences through the aforementioned training process. This allows the user to input descriptive text and random noise, and the video generation model to output a 360° video frame sequence. Therefore, after model training is complete, random noise and descriptive text related to the 3D scene to be generated can be input to generate a 360° video frame sequence. During training, a global keyframe attention mechanism is employed, introducing global context aggregation of keyframes to achieve efficient long-distance dependency calculation. A temporal attention mechanism is also introduced, using sample text features and sample video frame feature sequences processed by cross-attention to perform LoRA training on the model. Compared with existing technologies, the global keyframe attention mechanism can provide global semantic consistency while ensuring computational efficiency, ensuring the coherence of the 360° scene across different frames and avoiding structural inconsistencies, local geometric distortions, and missing details in the final generated 3D scene image. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the video generation model training method provided by the present invention.

[0017] Figure 2 This is a flowchart illustrating the process of combining the model structure in the video generation model training method provided by the present invention.

[0018] Figure 3 This is a comparison chart showing the effect of the target 360° video frame sequence output under different video generation model structures trained by the video generation model training method provided by this invention.

[0019] Figure 4 This is a flowchart illustrating the 3D scene generation method provided by the present invention.

[0020] Figure 5 This is a schematic diagram of the structure of the 3D scene generation device provided by the present invention.

[0021] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] The video generation model training method of the present invention, as described in the embodiments of the present invention, is as follows: Figure 1 As shown, the video generation model includes an image encoder, a text encoder, and a main network, which is a network based on stable diffusion. The training method includes steps S110 to S160.

[0024] Step S110: Obtain the sample 360° video frame sequence and the corresponding sample description text. Specifically, the panoramic image (i.e., the 360° panoramic image) can be projected to obtain multiple images, each of which is a frame in the sample 360° video frame sequence. The higher the projection density, the more images are obtained. The sample description text can be a description of the panoramic image, or it can be generated by annotating each frame in the sample 360° video frame sequence. For example: The panoramic image is densely and equidistantly projected to obtain 80 frames of panoramic images with uniform viewing angle changes, forming a 360° sample video frame sequence. The size of this video frame sequence is 80*512*512*3, and each frame represents a 4.5-degree viewing angle change.

[0025] Step S120: Input the sample 360° video frame sequence and the corresponding sample description into the image encoder and the text encoder respectively to obtain the sample video frame feature sequence and the corresponding sample text feature output by the image encoder and the text encoder respectively, and add random noise labels to the sample video frame feature sequence.

[0026] Specifically, such as Figure 2 As shown, the image encoder can be a VAE Encoder (Variational AutoEncoder). The image encoder extracts features from each video frame in the 360° video frame sequence to obtain a sample video frame feature sequence. For example, the VAE Encoder compresses an 80*512*512*3 video frame sequence into a sample video frame feature sequence (latents), obtaining an 80*64*64*4 sample video frame feature sequence. The text encoder can be a CLIP (Contrastive Language-Image Pre-Training). The text encoder extracts features from the sample descriptive text to obtain sample text features.

[0027] The random noise label is obtained by random sampling from white noise. Specifically, the random noise can be added to the sample video frame feature sequence using the native sampling method of stable diffusion 2.1. For each sample video frame feature in the sample video frame feature sequence, a random noise label is sampled using stable diffusion 2.1 and added to each sample video frame feature. The random noise label added to each sample video frame feature is different. The random noise label is a matrix with the same dimension as each sample video frame feature. Adding a random noise label is the sum of the sample video frame feature matrix and the random noise label noise matrix.

[0028] Step S130: Input the noise-added sample video frame feature sequence into the residual network module and self-attention module of the main network for processing.

[0029] The main network can be formed by cascading one or more UNets, each UNet containing a residual network module and a self-attention module. The feature sequence of sample video frames is first processed by the residual network module, and then by the self-attention module to obtain the processed feature sequence of sample video frames. For example, 80*64*64*4 features are noise-added and then enter the UNet network. They first pass through the residual network module, and then the 80 noisy frames of features are input into the self-attention module for self-attention operation.

[0030] Step S140: Determine multiple video frames with non-overlapping viewpoints in the sample video frame feature sequence processed by the residual network module and the self-attention module as key frames, and perform cross-attention processing on each key frame.

[0031] Specifically, in this step, a Global Keyframe Attention (GKA) module is added to the UNet network. GKA is essentially a cross-attention module. The purpose of this step is to obtain a longer receptive field with very little computational cost.

[0032] For example, regarding the 80 sample video frame features mentioned above, since each frame represents 4.5 degrees, the viewpoints of frames 1, 20, 40, and 60 do not overlap at all and can represent the entire 360-degree content. In this step, these 4 key frames are selected for cross-attention. The features of these 4 frames after cross-attention are added to the features of the original sample video frames, still resulting in 80 frames of features. This approach ensures computational efficiency while providing global semantic consistency, making the features in the sample video frame feature sequence coherent. This ensures the coherence of the final 360° scene between adjacent video frames, avoiding structural inconsistencies, local geometric distortions, and missing details in 3D scene images.

[0033] Step S150: Input the sample text features and the sample video frame feature sequence after cross-attention processing into the temporal attention module of the main network to train the parameters of LoRA (Low-Rank Adaptation) to output predicted noise. It should be noted that the temporal attention module is a temporal Transformer, and LoRA is a low-rank adaptation technique used to fine-tune large models. It can achieve good training results with only a few parameters, thereby improving the training efficiency of the model.

[0034] Step S160: Substitute the predicted noise and random noise labels into the loss function. The model training is completed when the loss function converges. Specifically, the loss function is a stable diffusion paradigm loss function.

[0035] The trained video generation model is used to output a 360° video frame sequence of the target based on the input target random noise and the descriptive text corresponding to the 3D scene to be generated.

[0036] In the video generation model training method of this embodiment, the above training process establishes the correlation between noise, descriptive text, and 360° video frame sequence. This allows the user to input descriptive text and random noise, and the video generation model to output a 360° video frame sequence. Therefore, after model training is complete, random noise and descriptive text related to the 3D scene to be generated can be input to generate a 360° video frame sequence. During training, a global keyframe attention mechanism is used, introducing global context aggregation of keyframes to achieve efficient long-distance dependency calculation. A temporal attention mechanism is also introduced, using sample text features and sample video frame feature sequences processed by cross-attention to perform LoRA training on the model. Compared with existing technologies, the global keyframe attention mechanism can provide global semantic consistency while ensuring computational efficiency, ensuring the continuity of the 360° scene across different frames, and avoiding structural inconsistencies, local geometric distortions, and missing details in the final generated 3D scene image.

[0037] like Figure 3 As shown, the output 360° video frame sequence of the video generation model trained by the video generation model training method provided in this embodiment is displayed (only four consecutive video frames are shown), where the red box indicates the intermediate transition frame. Figure 3 (a) indicates only when Figure 2 When the Global Keyframe Attention (GKA) module is trained after the self-attention module in the first case, the geometric process between the four consecutive frames generated is discontinuous. (b) indicates that only when the self-attention module is trained is the geometric process between the frames discontinuous. Figure 2 When a temporal attention module is trained after a self-attention module, artifacts will occur in the generated frames due to the lack of global context. (c) indicates that in Figure 2 When the self-attention module is trained followed by the Global Keyframe Attention (GKA) and Temporal Attention modules, the generated 360° video frame sequence has no artifacts in each frame and the geometric transitions are continuous. This makes the final 3D scene image generated from the 360° video frame sequence structurally consistent and free from local geometric distortion and loss of detail.

[0038] Since the video generation model generates a 360° video frame sequence of the target based on the descriptive text input by the user, the training focuses on processing the sample text features and the sample video frame feature sequence after cross-attention processing. Therefore, during the training phase, only the temporal attention module and the previous global keyframe attention mechanism module in the main network can be trained. The model parameters of the other modules are locked during training, and only the model parameters of the temporal attention module and the global keyframe attention mechanism module are updated, thereby speeding up model training and improving training efficiency.

[0039] In some embodiments, step S150 specifically includes: the temporal attention module performs sliding window processing on the sample video frame feature sequence after cross-attention processing using n video frame features as a sliding window. The sliding window processing involves performing temporal attention processing on the sample video frame features and the sample text features within the sliding window, outputting predicted noise based on the temporally attention-processed sample video frame feature sequence, and training the parameters of LoRA. Specifically, the predicted noise and random noise labels are substituted into the loss function to train the parameters of LoRA. It should be noted that both the predicted noise and the random noise labels are matrices with the same dimension as the sample video frame features.

[0040] In the sample video frame feature sequence after cross-attention processing, the n video frame features in the previous window and the n video frame features in the next window have m overlapping video frame features, where m is less than n and n is less than the total number of video frames in the sample 360° video frame sequence. The average feature is calculated for the m overlapping video frame features and used to replace the m overlapping video frame features to maintain the geometric consistency between the video frame features of adjacent window segments, so that the geometric continuity of the 360° video frame sequence generated by the finally trained video generation model is better.

[0041] Specifically, existing temporal attention modules in video models can only handle very short frames. In this embodiment, the temporal attention module uses a sliding window time transformer (SWTT) to handle long sequences of frames, such as 80 frames. Specifically, the temporal attention module processes only 10 (n=10) frames at a time, but there is some overlap in each processing step. For example, the first segment is frames 1-10, and the second segment is frames 6-15, with a sliding distance of 5 frames (m=5) each time. The features of the 5 overlapping frames are averaged to maintain geometric consistency between the video frame features of adjacent window segments. To train this sliding window, LoRA parameters are added to the temporal attention module. This is an efficient way to train the model; it can be considered as adding a small number of parameters to the original temporal attention layer for efficient training.

[0042] In this embodiment, the temporal attention module uses a sliding window time transformer (SWTT) to segment the 360° video sequence into overlapping time periods, which optimizes the temporal prior of the traditional video generation model. This design allows the video generation model to maintain local temporal consistency and geometric consistency when dealing with large viewpoint changes, overcoming the geometric distortion problem of existing technologies under large viewpoint changes.

[0043] It is understood that the main network of the video generation model in this embodiment is based on a stable diffusion network. This embodiment adds a global keyframe attention mechanism (GKA) module to the existing stable diffusion main network, and adopts a sliding window processing mechanism and adds LoRA parameters to the original temporal attention module of the stable diffusion main network. The improved stable diffusion main network generates a 360° video frame sequence. The 3D scene image generated by this 360° video frame sequence is structurally consistent and does not have the problems of local geometric distortion and lack of detail.

[0044] In some embodiments, step S110 specifically includes the following steps: Step 1: Project the panoramic image to obtain the sample 360° video frame sequence. Specifically, perform dense equidistant projection on the panoramic image to obtain a multi-frame panoramic image with uniform viewing angle changes, forming a 360° panoramic video sequence, i.e., the sample 360° video frame sequence.

[0045] Step 2: Input the sample 360° video frame sequence into the text annotation model to generate sub-description text corresponding to each video frame. In this step, the text annotation model can be BLIP2, which automatically annotates each video frame and generates sub-description text corresponding to each video frame.

[0046] Step 3: Input the multiple sub-description texts into the large model to obtain the sample description text output by the large model. In this step, the semantic analysis and summarization capabilities of the large model are used to analyze and summarize each sub-description text, generating sample description text corresponding to the 360° video frame sequence of the sample, which also corresponds to the panoramic image.

[0047] In this embodiment, a text annotation model is used to automatically annotate each frame in the 360° video frame sequence of the sample, generating sub-description text. Then, a large model is used to generate sample description text corresponding to the 360° video frame sequence of the sample based on each sub-description text, thereby realizing the automatic generation of sample description text.

[0048] This invention also provides a 3D scene generation method, such as... Figure 4 As shown, it includes steps S410 and S420.

[0049] Step S410: Input the target random noise and the target description text corresponding to the 3D scene to be generated into the video generation model to obtain the target 360° video frame sequence output by the video generation model. The video generation model is trained using the video generation model training method described in any of the above embodiments.

[0050] Step S420: Reconstruct the target 360° video frame sequence to generate a 3D scene. Specifically, the target 360° video frame sequence can be reconstructed based on DUST3R and 3D GS to obtain the 3D scene.

[0051] In the 3D scene generation method of this embodiment, since the video generation model is obtained by training the video generation model in the above embodiment, the target 360° video frame sequence generated by the video generation model has good geometric coherence or structural consistency between adjacent video frames, thus avoiding the problems of structural inconsistency, local geometric distortion and lack of detail in the final generated 3D scene image.

[0052] In some embodiments, the video generation model further includes: an image decoder, which is a decoder VAE Dncoder corresponding to the image encoder. Based on this, step S310 includes: The target description text is input into the text encoder to obtain the target text features output by the text encoder.

[0053] The target random noise and the target text features are input into the main network of the video generation model to obtain the target video frame feature sequence output by the main network. Since the video generation model trained using the above embodiment establishes the correlation between noise, descriptive text, and the 360° video frame sequence during training, in this step, by inputting the target random noise and the target text features, the video generation model can generate the corresponding target video frame feature sequence.

[0054] The target video frame feature sequence is input into the image decoder to obtain the target 360° video frame sequence output by the image decoder.

[0055] The 3D scene generation apparatus provided by the present invention is described below. The 3D scene generation apparatus described below and the 3D scene generation method described above can be referred to in correspondence.

[0056] The present invention also provides a 3D scene generation device, such as... Figure 5 As shown, it includes the following modules: The model calling module 510 is used to input the target random noise and the target description text corresponding to the 3D scene to be generated into the video generation model to obtain the target 360° video frame sequence output by the video generation model.

[0057] The 3D scene reconstruction module 520 is used to reconstruct the target 360° video frame sequence to generate a 3D scene.

[0058] The video generation model is trained based on the video generation model training method described above.

[0059] In some embodiments, the video generation model further includes: an image decoder; the model invocation module 510 is specifically used for: The target description text is input into the text encoder to obtain the target text features output by the text encoder.

[0060] The target random noise and the target text features are input into the main network of the video generation model to obtain the target video frame feature sequence output by the main network.

[0061] The target video frame feature sequence is input into the image decoder to obtain the target 360° video frame sequence output by the image decoder.

[0062] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a video generation model training method. The video generation model includes: an image encoder, a text encoder, and a main network. The method includes the following steps: Obtain the 360° video frame sequence of the sample and the corresponding sample description text.

[0063] The sample 360° video frame sequence and the corresponding sample description are input into the image encoder and the text encoder, respectively, to obtain the sample video frame feature sequence and the corresponding sample text feature output by the image encoder and the text encoder, respectively, and random noise labels are added to the sample video frame feature sequence.

[0064] The noisy sample video frame feature sequence is input into the residual network module and self-attention module of the main network for processing.

[0065] Multiple video frames with non-overlapping viewpoints in the sample video frame feature sequence processed by the residual network module and the self-attention module are identified as key frames, and cross-attention processing is performed on each key frame.

[0066] The sample text features and the sample video frame feature sequence after cross-attention processing are input into the temporal attention module of the main network to train the parameters of LoRA, so as to output the predicted noise.

[0067] The predicted noise and random noise labels are substituted into the loss function, and the model training is completed when the loss function converges.

[0068] The trained video generation model is used to output a 360° video frame sequence of the target based on the input target random noise and the descriptive text corresponding to the 3D scene to be generated.

[0069] Alternatively, execute a 3D scene generation method, which includes: The target random noise and the target description text corresponding to the 3D scene to be generated are input into the video generation model to obtain the target 360° video frame sequence output by the video generation model.

[0070] The target 360° video frame sequence is reconstructed to generate a 3D scene.

[0071] The video generation model is trained based on the video generation model training method described above.

[0072] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0073] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the video generation model training method provided by the above methods. The video generation model includes an image encoder, a text encoder, and a main network. The method includes the following steps: Obtain the 360° video frame sequence of the sample and the corresponding sample description text.

[0074] The sample 360° video frame sequence and the corresponding sample description are input into the image encoder and the text encoder, respectively, to obtain the sample video frame feature sequence and the corresponding sample text feature output by the image encoder and the text encoder, respectively, and random noise labels are added to the sample video frame feature sequence.

[0075] The noisy sample video frame feature sequence is input into the residual network module and self-attention module of the main network for processing.

[0076] Multiple video frames with non-overlapping viewpoints in the sample video frame feature sequence processed by the residual network module and the self-attention module are identified as key frames, and cross-attention processing is performed on each key frame.

[0077] The sample text features and the sample video frame feature sequence after cross-attention processing are input into the temporal attention module of the main network to train the parameters of LoRA, so as to output the predicted noise.

[0078] The predicted noise and random noise labels are substituted into the loss function, and the model training is completed when the loss function converges.

[0079] The trained video generation model is used to output a 360° video frame sequence of the target based on the input target random noise and the descriptive text corresponding to the 3D scene to be generated.

[0080] Alternatively, execute a 3D scene generation method, which includes: The target random noise and the target description text corresponding to the 3D scene to be generated are input into the video generation model to obtain the target 360° video frame sequence output by the video generation model.

[0081] The target 360° video frame sequence is reconstructed to generate a 3D scene.

[0082] The video generation model is trained based on the video generation model training method described above.

[0083] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the video generation model training method provided by the methods described above. The video generation model includes an image encoder, a text encoder, and a main network. The method includes the following steps: Obtain the 360° video frame sequence of the sample and the corresponding sample description text.

[0084] The sample 360° video frame sequence and the corresponding sample description are input into the image encoder and the text encoder, respectively, to obtain the sample video frame feature sequence and the corresponding sample text feature output by the image encoder and the text encoder, respectively, and random noise labels are added to the sample video frame feature sequence.

[0085] The noisy sample video frame feature sequence is input into the residual network module and self-attention module of the main network for processing.

[0086] Multiple video frames with non-overlapping viewpoints in the sample video frame feature sequence processed by the residual network module and the self-attention module are identified as key frames, and cross-attention processing is performed on each key frame.

[0087] The sample text features and the sample video frame feature sequence after cross-attention processing are input into the temporal attention module of the main network to train the parameters of LoRA, so as to output the predicted noise.

[0088] The predicted noise and random noise labels are substituted into the loss function, and the model training is completed when the loss function converges.

[0089] The trained video generation model is used to output a 360° video frame sequence of the target based on the input target random noise and the descriptive text corresponding to the 3D scene to be generated.

[0090] Alternatively, execute a 3D scene generation method, which includes: The target random noise and the target description text corresponding to the 3D scene to be generated are input into the video generation model to obtain the target 360° video frame sequence output by the video generation model.

[0091] The target 360° video frame sequence is reconstructed to generate a 3D scene.

[0092] The video generation model is trained based on the video generation model training method described above.

[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a video generation model, characterized in that, The video generation model includes an image encoder, a text encoder, and a main network, and the training method includes: Obtain the 360° video frame sequence of the sample and the corresponding sample description text; The sample 360° video frame sequence and the corresponding sample description are input into the image encoder and the text encoder respectively to obtain the sample video frame feature sequence and the corresponding sample text feature output by the image encoder and the text encoder respectively, and random noise labels are added to the sample video frame feature sequence. The noisy sample video frame feature sequence is input into the residual network module and self-attention module of the main network for processing; Multiple video frames with non-overlapping viewpoints in the sample video frame feature sequence processed by the residual network module and the self-attention module are identified as key frames, and cross-attention processing is performed on each key frame. The sample text features and the sample video frame feature sequence after cross-attention processing are input into the temporal attention module of the main network to train the parameters of LoRA, so as to output the predicted noise; Substitute the predicted noise and random noise labels into the loss function, and the model training is complete when the loss function converges. The trained video generation model is used to output a 360° video frame sequence of the target based on the input target random noise and the descriptive text corresponding to the 3D scene to be generated.

2. The video generation model training method according to claim 1, characterized in that, The step of inputting the sample text features and the sample video frame feature sequence after cross-attention processing into the temporal attention module of the main network to train the parameters of LoRA to output prediction noise includes: The temporal attention module uses n video frame features as a sliding window to process the sample video frame feature sequence after cross-attention processing. The sliding window processing involves performing temporal attention processing on the sample video frame features and the sample text features in the sliding window, outputting prediction noise based on the sample video frame feature sequence after temporal attention processing, and training the parameters of LoRA. In the sample video frame feature sequence after cross-attention processing, the n video frame features in the previous window and the n video frame features in the next window have m overlapping video frame features, where m is less than n and n is less than the total number of video frames in the sample 360° video frame sequence. The average feature is calculated for the m overlapping video frame features and used to replace the m overlapping video frame features.

3. The video generation model training method according to claim 1, characterized in that, The loss function is a loss function in the stablediffusion paradigm.

4. The video generation model training method according to any one of claims 1 to 3, characterized in that, Obtain the 360° video frame sequence of the sample and the corresponding sample description text, including: The 360° video frame sequence of the sample is obtained by projecting the panoramic image. Input the 360° video frame sequence of the sample into the text annotation model to generate sub-description text corresponding to each video frame; Multiple sub-description texts are input into a large model to obtain the sample description text output by the large model.

5. A method for generating a 3D scene, characterized in that, include: Input the target random noise and the target description text corresponding to the 3D scene to be generated into the video generation model to obtain the target 360° video frame sequence output by the video generation model; The target 360° video frame sequence is reconstructed to generate a 3D scene; The video generation model is trained based on the video generation model training method according to any one of claims 1 to 4.

6. The 3D scene generation method according to claim 5, characterized in that, The video generation model further includes: an image decoder; the step of inputting the target random noise and the target description text corresponding to the 3D scene to be generated into the video generation model to obtain the target 360° video frame sequence output by the video generation model includes: The target description text is input into the text encoder to obtain the target text features output by the text encoder; The target random noise and the target text features are input into the main network of the video generation model to obtain the target video frame feature sequence output by the main network. The target video frame feature sequence is input into the image decoder to obtain the target 360° video frame sequence output by the image decoder.

7. A 3D scene generation device, characterized in that, include: The model calling module is used to input the target random noise and the target description text corresponding to the 3D scene to be generated into the video generation model to obtain the target 360° video frame sequence output by the video generation model. The 3D scene reconstruction module is used to reconstruct the target 360° video frame sequence to generate a 3D scene; The video generation model is trained based on the video generation model training method according to any one of claims 1 to 4.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the video generation model training method as described in any one of claims 1 to 4, or the 3D scene generation method as described in claim 5 or 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the video generation model training method as described in any one of claims 1 to 4, or the 3D scene generation method as described in claim 5 or 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the video generation model training method as described in any one of claims 1 to 4, or the 3D scene generation method as described in claim 5 or 6.

Images