Video data set construction method and device, model training method and device and video generation method and device

Abstract

One or more embodiments of the invention provide a video data set construction method, a model training method, and a video generation method and device. According to the video data set construction method, a large number of illumination tracks are constructed in a 3D illumination network, and then a video of a whiteboard image representing illumination information and a model video representing a 3D model corresponding to each illumination track are generated. Then, a video data set for training a text video generation model is generated based on the video characterizing the illumination information and the video characterizing the model information.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of machine vision technology, and in particular, to a video data set construction method, a model training method, and a video generation method and device. Background Art

[0002] Lighting plays a pivotal role in ensuring the naturalness of video generation and has a significant impact on the aesthetic quality of generated videos. However, due to the lack of well-annotated training datasets that show lighting changes and can well define lighting information, it is impossible to further obtain a video generation model that can generate videos with good lighting effects.

[0003] Therefore, it is necessary to provide a video dataset construction method to obtain a video dataset that can better express lighting information, and then train a video generation model that can generate videos that express lighting effects well. Summary of the invention

[0004] In order to obtain a video dataset that can better express lighting information, and then train a video generation model that can generate videos that express lighting effects well, one or more embodiments of this specification provide a video dataset construction method, a model training method, a video generation method and device.

[0005] In a first aspect, one or more embodiments of the present specification provide a method for constructing a video dataset, the method comprising: constructing multiple lighting trajectories in a 3D lighting grid; the 3D lighting grid is a three-dimensional grid in which grid points are used as lighting positions; the lighting trajectory is a trajectory connected by multiple grid points; obtaining a first image set and at least one second image set corresponding to each lighting trajectory; the first image set includes a whiteboard image generated by a point light source being located at each lighting position of the corresponding lighting trajectory to illuminate a whiteboard in a blank space; the second image set includes a model image generated by a point light source being located at each lighting position of the corresponding lighting trajectory to illuminate a 3D model; the second image set corresponds to the 3D model in a one-to-one manner; generating a first video corresponding to each first image set and a second video corresponding to each second image set; annotating text information for each second video; constructing a video dataset for training a text video generation model based on all first videos and all second videos annotated with text information; the video dataset includes multiple training samples, each training sample includes a first video corresponding to a lighting trajectory, and a second video annotated with text information of a 3D model corresponding to the lighting trajectory, and both the lighting trajectory and the 3D model correspond to the training samples in a one-to-one manner.

[0006] In one possible implementation, the first lighting trajectory is any one of the multiple lighting trajectories; obtaining a first image set corresponding to the first lighting trajectory includes: controlling a point light source to move sequentially at each lighting position of the first lighting trajectory, and after the point light source moves to any lighting position to illuminate a whiteboard in a blank space, generating a frame of whiteboard image by rendering through a virtual engine; constructing all whiteboard images corresponding to the first lighting trajectory into a first image set corresponding to the first lighting trajectory in chronological order of generation time.

[0007] In a possible implementation, the first lighting trajectory is any one of the multiple lighting trajectories; the third image set is any one of the at least one second image set corresponding to the first lighting trajectory; the first 3D model is the 3D model corresponding to the third image set; obtaining the third image set corresponding to the first lighting trajectory includes: controlling a point light source to move sequentially on each lighting position of the first lighting trajectory, and after the point light source moves to any lighting position to illuminate the first 3D model, generating a frame of model image through virtual engine rendering; constructing all model images corresponding to the first lighting trajectory and the first 3D model into a third image set corresponding to the first lighting trajectory and the first 3D model in chronological order of generation time.

[0008] In one possible implementation, text information is annotated for each second video, including: generating initial text information for each second video through an image language pre-training model; performing title enhancement processing on the initial text information of each second video to generate enhanced text information for each second video; and annotating corresponding enhanced text information for each second video.

[0009] In a possible implementation, the 3D model is a 3D character model.

[0010] In a second aspect, one or more embodiments of the present specification also provide a model training method based on a video dataset, the model training method comprising: constructing a frozen denoising network and an external transformer network; the frozen denoising network has the same architecture as a pre-trained text video generation model, and is used to freeze the network weight parameters of the video denoising network in the pre-trained text video generation model; the video denoising network comprises a variational self-attention encoder for feature extraction and a preset number of diffusion transformer blocks; the external transformer network comprises the same variational self-attention encoder as the video denoising network, and the preset number of illumination encoder blocks, wherein the illumination encoder block is a self-attention layer Transformer architecture; based on the training samples and objective function in the video data set, the pre-trained text video generation model, the external transformer network and the frozen denoising network are trained to generate a trained text video generation model and an external transformer network; the objective function is a decoupling loss function of the predicted noise of the video denoising network and the predicted noise of the frozen denoising network; wherein, in any training iteration process, the illumination features extracted from each illumination encoder block are input into the corresponding diffusion transformer block, and the features are fused with the video features extracted from the diffusion transformer block and then input into the multi-head attention layer of the diffusion transformer block.

[0011] In one possible implementation, the pre-trained text video generation model, the external transformer network and the frozen denoising network are trained based on the training samples and the objective function in the video data set, including: inputting the second video with text information annotated in the training samples in the video data set into the pre-trained text video generation model and the frozen denoising network respectively, and inputting the first video in the corresponding training sample into the external transformer network, and performing model training based on the objective function.

[0012] In a possible implementation, the lighting feature and the video feature are fused by element-by-element addition.

[0013] In a third aspect, one or more embodiments of the present specification also provide a video generation method based on an external transformer network and a text video generation model trained and generated by the model training method of the second aspect, the video generation method comprising: selecting a target lighting trajectory required to generate a target video from a plurality of lighting trajectories in a 3D lighting grid; inputting a first video corresponding to the target lighting trajectory into the external transformer network, and inputting target text information used to generate the target video into the text video generation model to generate the target video; wherein, in the process of generating the target video, the lighting features extracted from each lighting encoder block are input into the corresponding diffusion transformer block, and the lighting features extracted from the diffusion transformer block are feature fused and then input into the multi-head attention layer of the diffusion transformer block.

[0014] In a fourth aspect, one or more embodiments of the present specification further provide a video data set construction device, the device comprising: a first construction module, configured to construct a plurality of lighting tracks in a 3D lighting grid; the 3D lighting grid is a three-dimensional grid in which grid points are used as lighting positions; the lighting track is a track formed by connecting a plurality of the grid points; an acquisition module, configured to acquire a first image set and at least one second image set corresponding to each lighting track; the first image set comprises a whiteboard image corresponding to a whiteboard in a blank space illuminated by a point light source at each lighting position of the corresponding lighting track; the second image set comprises a whiteboard image corresponding to a whiteboard in a blank space illuminated by the point light source at each lighting position of the corresponding lighting track; The model image is generated corresponding to the model; the second image set corresponds to the 3D model one-to-one; a first generation module is used to generate a first video corresponding to each first image set and a second video corresponding to each second image set; a labeling module is used to label text information for each second video; a second construction module is used to construct a video data set for training a text video generation model based on all first videos and all second videos labeled with text information; the video data set includes multiple training samples, each training sample includes a first video corresponding to a lighting trajectory, and a second video corresponding to a 3D model of the lighting trajectory labeled with text information, and the lighting trajectory and the 3D model both correspond to the training samples one-to-one.

[0015] In a possible implementation, the first lighting trajectory is any one of the multiple lighting trajectories; the acquisition module is used to acquire a first image set corresponding to the first lighting trajectory, including: the acquisition module is used to: control the point light source to move in sequence at each lighting position of the first lighting trajectory, and after the point light source moves to any lighting position to illuminate the whiteboard in the blank space, generate a frame of whiteboard image through virtual engine rendering; all whiteboard images corresponding to the first lighting trajectory are constructed into a first image set corresponding to the first lighting trajectory in chronological order of generation time.

[0016] In a possible implementation, the first lighting trajectory is any one of the multiple lighting trajectories; the third image set is any one of the at least one second image set corresponding to the first lighting trajectory; the first 3D model is the 3D model corresponding to the third image set; the acquisition module is used to acquire the third image set corresponding to the first lighting trajectory, including: the acquisition module is used to: control the point light source to move in sequence on each lighting position of the first lighting trajectory, after the point light source moves to any lighting position to illuminate the first 3D model, generate a frame of model image through virtual engine rendering; all model images corresponding to the first lighting trajectory and the first 3D model are constructed into a third image set corresponding to the first lighting trajectory and the first 3D model in chronological order of generation time.

[0017] In one possible implementation, the annotation module is used to annotate text information for each second video, including: the annotation module is used to: generate initial text information for each second video through an image language pre-training model; perform title enhancement processing on the initial text information of each second video to generate enhanced text information for each second video; and annotate corresponding enhanced text information for each second video.

[0018] In a possible implementation, the 3D model is a 3D character model.

[0019] In a fifth aspect, one or more embodiments of the present specification further provide a model training device based on a video dataset, wherein the video dataset is constructed by the video dataset construction device of the fourth aspect, and the model training device comprises: a third construction module, used to construct a frozen denoising network and an external transformer network; the frozen denoising network has the same architecture as the pre-trained text video generation model, and is used to freeze the network weight parameters of the video denoising network in the pre-trained text video generation model; the video denoising network comprises a variational self-decomposition encoder for feature extraction and a preset number of diffusion transformer blocks; the external transformer network comprises the same variational self-decomposition encoder as the video denoising network, and the preset number of illumination encoder blocks, The illumination encoder block is a transformer architecture including a self-attention layer; a training module is used to train the pre-trained text video generation model, the external transformer network and the frozen denoising network based on the training samples and the objective function in the video data set to generate a trained text video generation model and an external transformer network; the objective function is a decoupling loss function of the predicted noise of the video denoising network and the predicted noise of the frozen denoising network; wherein, in any training iteration process, the illumination features extracted from each illumination encoder block are input into the corresponding diffusion transformer block, and are input into the multi-head attention layer of the diffusion transformer block after feature fusion with the video features extracted from the diffusion transformer block.

[0020] In one possible implementation, the training module is used to train the pre-trained text video generation model, the external transformer network and the frozen denoising network based on the training samples and the objective function in the video data set, including: the training module is used to: input the second video with text information annotated in the training samples in the video data set into the pre-trained text video generation model and the frozen denoising network respectively, and input the first video in the corresponding training sample into the external transformer network, and perform model training based on the objective function.

[0021] In a possible implementation, the training module is used to perform feature fusion on the lighting feature and the video feature by element-by-element addition.

[0022] In a sixth aspect, one or more embodiments of the present specification also provide a video generation device based on an external transformer network and a text video generation model trained and generated by the model training device of the fifth aspect, the video generation device comprising: a selection module for selecting a target lighting trajectory required to generate a target video from a plurality of lighting trajectories in a 3D lighting grid; a second generation module for inputting a first video corresponding to the target lighting trajectory into the external transformer network, and inputting target text information used to generate the target video into the text video generation model to generate the target video; wherein, in the process of generating the target video, the lighting features extracted from each lighting encoder block are input into the corresponding diffusion transformer block, and are input into the multi-head attention layer of the diffusion transformer block after feature fusion with the video features extracted by the diffusion transformer block.

[0023] In the seventh aspect, one or more embodiments of the present specification also provide an electronic device, comprising a memory and a processor; the memory is used to store a computer program product; the processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the method of the first aspect, the second aspect or the third aspect mentioned above is implemented.

[0024] In an eighth aspect, one or more embodiments of the present specification further provide a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed, the method of the first aspect, the second aspect, or the third aspect described above is implemented.

[0025] In summary, in order to obtain a video dataset that can better express lighting information, and then train a text video generation model to generate a video that expresses lighting effects well, one or more embodiments of this specification provide a video dataset construction method, a model training method, a video generation method and a device. In the video dataset construction method, a large number of lighting trajectories are constructed in a 3D lighting network, and then a video of a whiteboard image representing the lighting information and a model video representing a 3D model (such as a 3D character model) corresponding to each lighting trajectory are generated. Afterwards, a video dataset for training a text video generation model is generated based on the video representing the lighting information and the video representing the model information.

[0026] When training the text generation model based on the above video dataset, the lighting information can be integrated into the text video generation model. In the subsequent video generation process, the lighting direction and motion trajectory can be controlled so that the generated video can better express the lighting effect, better meet the needs of users, and provide a better user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of one or more embodiments of the present specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of one or more embodiments of the present specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A flowchart of a method for constructing a video data set provided in one or more embodiments of this specification;

[0029] Figure 2 A schematic diagram of an application scenario provided for one or more embodiments of this specification;

[0030] Figure 3 A flowchart of a model training method provided for one or more embodiments of this specification;

[0031] Figure 4 A schematic diagram of another application scenario provided for one or more embodiments of this specification;

[0032] Figure 5 A flowchart of a video generation method provided for one or more embodiments of this specification;

[0033] Figure 6 A structural block diagram of a video data set construction device provided in one or more embodiments of this specification;

[0034] Figure 7 A structural block diagram of a model training device provided for one or more embodiments of this specification;

[0035] Figure 8 A structural block diagram of a video generation device provided for one or more embodiments of this specification;

[0036] Fig. 9 A structural block diagram of an electronic device provided for one or more embodiments of this specification. DETAILED DESCRIPTION

[0037] The following further describes one or more embodiments of the present specification in detail through the accompanying drawings and examples. Through these descriptions, the features and advantages of one or more embodiments of the present specification will become clearer and more specific.

[0038] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0039] In addition, the technical features involved in different implementations of one or more embodiments of this specification described below can be combined with each other as long as they do not conflict with each other.

[0040] To facilitate understanding, the application scenarios of the technical solutions provided by one or more embodiments of this specification are first described below.

[0041] Lighting is an essential element in video generation. It is one of the factors that determine the overall aesthetic quality of a video, and is also an important means of conveying emotions, highlighting character features, and guiding viewers' attention. However, due to the scarcity of clearly labeled training datasets that show lighting changes and can well define lighting information, it is impossible to further obtain a video generation model that can generate videos with good lighting effects, or in other words, it is impossible to efficiently control lighting during the video generation process.

[0042] In order to obtain a video with good lighting effect expression, one or more embodiments of the present specification provide a video data set construction method, a model training method, a video generation method and a device.

[0043] See also Figure 1 , Figure 1 This is a flow chart of a method for constructing a video data set provided by one or more embodiments of this specification. The method can be applied to a server or a terminal device, etc. The following takes a terminal device as an example to introduce the content of the embodiment. Figure 1 As shown, the method may include the following steps:

[0044] Step S102: construct multiple lighting tracks in the 3D lighting grid.

[0045] The 3D lighting grid is a three-dimensional grid whose grid points are used as lighting positions. The lighting track is a track formed by connecting multiple grid points in the 3D lighting grid.

[0046] The size of the 3D lighting grid and the distance between the grid points in the 3D lighting grid can be set according to the needs of the actual application scenario. For example, the size of the 3D lighting grid can be set to 160cm (centimeter) × 160cm × 160cm. The grid points in the 3D lighting grid can be set to a uniformly spaced mode, and the distance between the grid points can be set to 5cm. Figure 2 As shown, the 3D lighting grid can be set to Figure 2 The three-dimensional grid shown.

[0047] The lighting tracks constructed in the 3D lighting grid can be set according to the requirements of the actual application scenario. For example, the lighting tracks can include horizontal lighting tracks along the horizontal direction, vertical lighting tracks along the vertical direction, diagonal lighting tracks along the diagonal of the 3D lighting grid, etc. The number of lighting tracks can also be set according to the requirements of the actual application scenario.

[0048] Step S104: Acquire a first image set and at least one second image set corresponding to each lighting track.

[0049] The first image set includes whiteboard images generated when point light sources are located at each lighting position of the corresponding lighting track to illuminate the whiteboard in the blank space. For example, the first lighting track is any one of the multiple lighting tracks constructed, and there are 10 grid points on the first lighting track. Then the first image set corresponding to the first lighting track includes whiteboard images generated when point light sources are located at each of the 10 grid points to illuminate the whiteboard. There are no other objects in the blank space except the whiteboard.

[0050] The second image set includes model images corresponding to the 3D model illuminated by point light sources at various lighting positions of the corresponding lighting trajectory. The second image set corresponds to the 3D model one by one. Exemplarily, the 3D model can be a 3D character model. It should be understood that the 3D model can also be a 3D model of an animal or other object, and the embodiments of this specification are not limited to this.

[0051] Optionally, the number of 3D models can be set according to the needs of the actual application scenario. The number of second image sets corresponding to each lighting track is the same as the number of 3D models. Exemplarily, when constructing a video data set, 65 3D character models can be set. Then, the number of second image sets corresponding to each lighting track is 65.

[0052] Exemplarily, taking the first illumination trajectory and any 3D model as an example, the corresponding second image set includes model images generated after the point light source is located at each of the above 10 grid points and illuminates the 3D model.

[0053] There may be multiple implementations for acquiring the first image set and at least one second image set corresponding to each lighting track. The following takes the first lighting track as an example to illustrate the implementation of acquiring the first image set and at least one second image set corresponding to each lighting track.

[0054] In some optional embodiments, obtaining the first image set corresponding to the first lighting track can be achieved in the following manner: controlling the point light source to move sequentially at each lighting position of the first lighting track, after the point light source moves to any lighting position of the first lighting track to illuminate the whiteboard in the blank space, a frame of whiteboard image is generated by rendering through a virtual engine; all whiteboard images corresponding to the first lighting track are constructed into a first image set corresponding to the first lighting track in the order of generation time. In this way, the first image set corresponding to the first lighting track can be directly and quickly obtained, and the efficiency of obtaining the first image set is high.

[0055] In some optional embodiments, obtaining the first image set corresponding to the first lighting trajectory can also be achieved in the following manner: controlling the point light source to move to each lighting position of the 3D lighting grid, after the point light source moves to any lighting position of the 3D lighting grid to illuminate the whiteboard in the blank space, a frame of whiteboard image is generated by virtual engine rendering, and the position information of the corresponding grid point is marked in each frame of the whiteboard image; then, based on the position information of the grid point, the whiteboard image corresponding to each grid point in the first lighting trajectory is selected from all the whiteboard images generated above, and then all the selected whiteboard images are constructed into the first image set corresponding to the first lighting trajectory according to the trajectory sequence of the first lighting trajectory. In this way, there is no need to repeatedly render and generate the same whiteboard image for the same lighting position, which can save the cost of rendering resources.

[0056] In some optional embodiments, obtaining the first image set corresponding to the first lighting trajectory can also be achieved in the following manner: controlling the point light source to move to each lighting position of each constructed lighting trajectory respectively, after the point light source moves to any lighting position of the lighting trajectory to illuminate the whiteboard in the blank space, a frame of whiteboard image is generated by virtual engine rendering, and the position information of the corresponding grid point is marked in each frame of the whiteboard image; then, based on the position information of the grid point, all generated whiteboard images are deduplicated, that is, only one frame of whiteboard images with the same lighting position is retained; then, the whiteboard image corresponding to each grid point in the first lighting trajectory is selected from all the deduplicated whiteboard images, and then, according to the trajectory order of the first lighting trajectory, all the selected whiteboard images are constructed into a first image set corresponding to the first lighting trajectory. In this way, there is no need to store a large number of identical whiteboard images, and storage resources can be optimized.

[0057] Similarly, the implementation method of obtaining any second image set corresponding to each lighting track may also include multiple methods. Assume that the first lighting track is any one of the multiple lighting tracks constructed. The third image set is any one of the at least one second image set corresponding to the first lighting track. The first 3D model is a 3D model corresponding to the third image set. Below, taking the first lighting track, the first 3D model and the third image set as examples, the implementation method of obtaining the second image set corresponding to each lighting track is described.

[0058] In some optional embodiments, obtaining the third image set corresponding to the first lighting trajectory can be achieved in the following manner: controlling the point light source to move sequentially at each lighting position of the first lighting trajectory, after the point light source moves to any lighting position of the first lighting trajectory to illuminate the first 3D model, a frame of model image is generated by virtual engine rendering; all model images corresponding to the first lighting trajectory and the first 3D model are constructed into a third image set corresponding to the first lighting trajectory and the first 3D model in the order of generation time. In this way, the third image set corresponding to the first lighting trajectory can be directly and quickly obtained, and the efficiency of obtaining the third image set is relatively high.

[0059] In some optional embodiments, obtaining the third image set corresponding to the first lighting trajectory can also be achieved in the following manner: controlling the point light source to move to each lighting position of the 3D lighting grid, after the point light source moves to any lighting position of the 3D lighting grid to illuminate the first 3D model, a frame of model image is generated by virtual engine rendering, and the position information of the corresponding grid point is marked in each frame of the model image; then, based on the position information of the grid point, the model image corresponding to each grid point in the first lighting trajectory is selected from all the model images generated above, and then all the selected model images are constructed into a third image set corresponding to the first lighting trajectory and the first 3D model according to the trajectory sequence of the first lighting trajectory. In this way, there is no need to repeatedly render and generate the same model image for the same lighting position, which can save the cost of rendering resources.

[0060] In some optional embodiments, obtaining the third image set corresponding to the first lighting trajectory can also be achieved in the following manner: controlling the point light source to move to each lighting position of the lighting trajectory according to each constructed lighting trajectory, after the point light source moves to any lighting position of the lighting trajectory to illuminate the first 3D model, a frame of model image is generated by virtual engine rendering, and the position information of the corresponding grid point is marked in each frame of the model image; then, based on the position information of the grid point, all the generated model images are deduplicated, that is, only one frame of model images with the same lighting position is retained; then, the model image corresponding to each grid point in the first lighting trajectory is selected from all the deduplicated model images, and then, according to the trajectory order of the first lighting trajectory, all the selected model images are constructed into a third image set corresponding to the first lighting trajectory and the first 3D model. In this way, there is no need to store a large number of identical template images, and storage resources can be optimized.

[0061] Step S106: Generate a first video corresponding to each first image set and a second video corresponding to each second image set.

[0062] After obtaining the first image set and at least one second image set corresponding to each lighting track, a video corresponding to each first image set is generated, recorded as a first video; and a video corresponding to each second image set is generated, recorded as a second video. In addition, each first video is associated with the corresponding lighting track. Each second video is associated with the corresponding lighting track and the 3D model.

[0063] Step S108: annotate text information for each second video.

[0064] In some optional embodiments, the initial text information of each second video can be generated by a bootstrapping language-image pre-training (BLIP) model, and then the corresponding initial text information is annotated for each second video, that is, the initial text information of each second video is annotated on the second video as the text information.

[0065] In some optional embodiments, the initial text information of each second video can be generated by an image language pre-training model; then, the initial text information of each second video is subjected to title enhancement processing to generate enhanced text information of each second video; and then, the corresponding enhanced text information is annotated for each second video. That is, the enhanced text information of each second video is annotated as text information on the second video. The text information annotated for the second video in this way is more accurate.

[0066] Step S110: construct a video dataset for training a text-video generation model based on all first videos and all second videos annotated with text information.

[0067] The video data set includes multiple training samples, each training sample includes a first video corresponding to a lighting trajectory, and a second video with annotated text information of a 3D model corresponding to the lighting trajectory, and the lighting trajectory and the 3D model correspond to the training sample one-to-one.

[0068] For example, if there are 65 3D character models in total, the first lighting trajectory may correspond to 65 training samples. Each training sample includes a first video corresponding to the first lighting trajectory and a second video corresponding to one of the 3D character models in the 65 second videos annotated with text information corresponding to the first lighting trajectory. Different training samples correspond to second videos of different 3D character models.

[0069] In the video dataset construction method provided in the above embodiment, a large number of lighting tracks are constructed in the 3D lighting network, and then a video of a whiteboard image representing lighting information and a model video representing a 3D model (such as a 3D character model) corresponding to each lighting track are generated. Afterwards, a video dataset for training a text video generation model is generated based on the video representing lighting information and the video representing model information.

[0070] When training the text generation model based on the above video dataset, the lighting information can be integrated into the text video generation model. In the subsequent video generation process, the lighting direction and motion trajectory can be controlled so that the generated video can better express the lighting effect, better meet the needs of users, and provide a better user experience.

[0071] It can be understood that the above embodiments are only examples and can be modified in actual implementation. Those skilled in the art can understand that the modification methods of the above embodiments without creative labor fall within the protection scope of one or more embodiments of this specification and will not be repeated in the embodiments.

[0072] Based on the same inventive concept, one or more embodiments of this specification further provide a model training method based on a video data set, wherein the video data set can be constructed according to the video data set construction method provided in the above embodiment.

[0073] See also Figure 3 , Figure 3 The present invention provides a flow chart of a model training method based on a video data set for one or more embodiments of the present invention. The method can be applied to a server or a terminal device, etc. The following takes a terminal device as an example to introduce the content of the embodiment. Figure 3 As shown, the method may include the following steps:

[0074] Step S202: construct a frozen denoising network and an external transformer network.

[0075] The frozen denoising network has the same architecture as the pre-trained text-video generation model, and is used to freeze the network weight parameters of the video denoising network in the pre-trained text-video generation model. That is, the pre-trained text-video generation model may include a video denoising network. The frozen denoising network can be set to the same architecture as the pre-trained text-video generation model, and then the network weight parameters of the video denoising network in the architecture are frozen.

[0076] The video denoising network in the pre-trained text video generation model may include a variational encoder for feature extraction and a preset number of diffusion transformer blocks. The preset number can be set according to the needs of the actual application scenario. For example, the preset number can be set to 32, that is, the video denoising network may include 32 diffusion transformer blocks. Each diffusion transformer block includes a multi-head attention layer.

[0077] The outer transformer network consists of the same variational self-attention encoder as the video denoising network, and a preset number of light encoder blocks, which are transformer architectures including self-attention layers.

[0078] For example, the architecture of the video denoising network and the external transformer network can be found in Figure 4 .

[0079] Step S204: Based on the training samples and the objective function in the video data set, the pre-trained text-video generation model, the external transformer network and the frozen denoising network are trained to generate a trained text-video generation model and an external transformer network.

[0080] Among them, the objective function is the decoupling loss function of the predicted noise of the video denoising network and the predicted noise of the frozen denoising network.

[0081] Exemplarily, the objective function may include the following:

[0082]

[0083] L total =L denoise +βL dis .

[0084] Among them, L dis represents the decoupling loss, L denoise represents the denoising loss, σ(·) represents the standard deviation, and μ(·) represents the mean. represents the predicted noise of the frozen denoising network at time step t. represents the predicted noise of the video denoising network with time step t. N represents the total step size. c t Indicates text condition. β is set to 3.0.

[0085] like Figure 4 As shown in the figure, during any training iteration, the lighting features extracted from each lighting encoder block in the external transformer network are input into the corresponding diffusion transformer block in the video denoising network, and then fused with the video features extracted from the diffusion transformer block and input into the multi-head attention layer of the diffusion transformer block. Thus, the lighting information is integrated into the video.

[0086] In some optional embodiments, the lighting features and the video features may be fused by element-by-element addition.

[0087] In some optional embodiments, when executing step S204, the second video with labeled text information in the training sample in the video dataset is input into the pre-trained text video generation model and the frozen denoising network respectively, and the first video in the corresponding training sample is input into the external transformer network, and the model training is performed based on the objective function.

[0088] It is assumed that the first training sample is any training sample in the video data set. During the training process, the second video with annotated text information included in the first training sample is input into the pre-trained text video generation model and the frozen denoising network respectively, and the first video included in the first training sample is input into the external transformer network, and then the predicted noise of the video denoising network and the predicted noise of the frozen denoising network are calculated, and the predicted noise of the video denoising network and the predicted noise of the frozen denoising network are used to optimize the objective function, and the model parameters are adjusted.

[0089] When the objective function is optimized to the minimum loss, the model training is completed, and the trained text-video generation model and external transformer network can be obtained.

[0090] In the model training method provided by the above embodiment, on the basis of the pre-trained text video generation model, a frozen denoising network with the same architecture as the pre-trained text video generation model is constructed, and the network weight parameters of the video denoising network are frozen in the frozen denoising network. An external transformer network that shares the same variable self-encoder and the same number of transformer blocks as the video denoising network is also constructed. During the model training process, the lighting features are integrated into the video features through the external transformer network, and the pre-trained text video generation model and the external transformer network (together as a control branch) are established through the frozen denoising network, and the decoupling between the two branches with the frozen denoising network (as a frozen branch) avoids the overfitting of the model, so that the training is more accurate, and the text video generation model and the external transformer network that can control the lighting direction and the lighting motion trajectory can be generated. Videos that are more in line with user needs can be generated later, and the user experience is better.

[0091] Based on the same inventive concept, one or more embodiments of this specification further provide a video generation method. In the video generation method, the text video generation model trained by the model training method provided by the above embodiment and the external transformer network are used to generate the target video required by the user.

[0092] See also Figure 5 , Figure 5 This is a flow chart of a video generation method provided by one or more embodiments of this specification. The method can be applied to a server or a terminal device, etc. The following takes the terminal device as an example to introduce the content of the embodiment. Figure 5 As shown, the method may include the following steps:

[0093] Step S302: Select a target lighting trajectory required for generating a target video from a plurality of lighting trajectories in the 3D lighting grid.

[0094] The specific contents of the 3D lighting grid and the lighting trajectory may refer to the contents of the aforementioned embodiments, which will not be described in detail here.

[0095] In some optional embodiments, a target lighting track required for generating a target video may be selected from a plurality of lighting tracks in a 3D lighting grid according to user input.

[0096] Step S304: input the first video corresponding to the target lighting trajectory into an external transformer network, and input the target text information used to generate the target video into a text video generation model to generate the target video.

[0097] In the process of generating the target video, the lighting features extracted from each lighting encoder block in the external transformer network are input into the corresponding diffusion transformer block of the video denoising network in the text video generation model, and are fused with the video features extracted by the diffusion transformer block and then input into the multi-head attention layer of the diffusion transformer block.

[0098] The method for acquiring the first video corresponding to the target lighting trajectory can refer to the content of the aforementioned embodiment, which will not be repeated here.

[0099] In some optional embodiments, the target text information may be obtained based on user input, or may be pre-stored in the system and retrieved from the system, which is not limited in the embodiments of this specification.

[0100] In the video generation method provided in the above embodiment, lighting information that meets user needs is integrated during the process of generating the video, so that the generated video can better express the lighting effect and provide a better user experience.

[0101] Based on the same inventive concept, one or more embodiments of the present specification also provide a video data set construction device, a model training device, and a video generation device. Since the principles of the problems solved by these devices are similar to those of the aforementioned methods, the implementation of the devices can refer to the implementation of the aforementioned methods, and the repeated parts will not be repeated.

[0102] See also Figure 6 , Figure 6 This is a structural block diagram of a video data set construction device provided by one or more embodiments of this specification. Figure 6 As shown, the video data set construction device 600 may include: a first construction module 601, an acquisition module 602, a first generation module 603, a labeling module 604, and a second construction module 605.

[0103] A first construction module 601 is used to construct a plurality of lighting tracks in a 3D lighting grid; the 3D lighting grid is a three-dimensional grid whose grid points are used as lighting positions; the lighting track is a track connected by a plurality of the grid points;

[0104] The acquisition module 602 is used to acquire a first image set and at least one second image set corresponding to each lighting track; the first image set includes whiteboard images generated when point light sources are located at each lighting position of the corresponding lighting track and illuminate a whiteboard in a blank space; the second image set includes model images generated when the point light sources are located at each lighting position of the corresponding lighting track and illuminate a 3D model; the second image set corresponds to the 3D model one by one;

[0105] A first generating module 603, configured to generate a first video corresponding to each first image set and a second video corresponding to each second image set;

[0106] Annotation module 604, used to annotate text information for each second video;

[0107] The second construction module 605 is used to construct a video dataset for training a text video generation model based on all the first videos and all the second videos annotated with text information; the video dataset includes multiple training samples, each training sample includes a first video corresponding to a lighting trajectory, and a second video annotated with text information of a 3D model corresponding to the lighting trajectory, and the lighting trajectory and the 3D model both correspond to the training samples one by one.

[0108] In a possible implementation, the first lighting trajectory is any one of the multiple lighting trajectories; the acquisition module 602 is used to acquire a first image set corresponding to the first lighting trajectory, including: the acquisition module 602 is used to: control the point light source to move in sequence at each lighting position of the first lighting trajectory, after the point light source moves to any lighting position to illuminate the whiteboard in the blank space, generate a frame of whiteboard image through virtual engine rendering; all whiteboard images corresponding to the first lighting trajectory are constructed into a first image set corresponding to the first lighting trajectory in chronological order of generation time.

[0109] In a possible implementation, the first lighting trajectory is any one of the multiple lighting trajectories; the third image set is any one of the at least one second image set corresponding to the first lighting trajectory; the first 3D model is the 3D model corresponding to the third image set; the acquisition module 602 is used to acquire the third image set corresponding to the first lighting trajectory, including: the acquisition module 602 is used to: control the point light source to move in sequence on each lighting position of the first lighting trajectory, after the point light source moves to any lighting position to illuminate the first 3D model, generate a frame of model image through virtual engine rendering; all model images corresponding to the first lighting trajectory and the first 3D model are constructed into a third image set corresponding to the first lighting trajectory and the first 3D model in chronological order of generation time.

[0110] In one possible implementation, the annotation module 604 is used to annotate text information for each second video, including: the annotation module 604 is used to: generate initial text information for each second video through an image language pre-training model; perform title enhancement processing on the initial text information of each second video to generate enhanced text information for each second video; and annotate corresponding enhanced text information for each second video.

[0111] In a possible implementation, the 3D model is a 3D character model.

[0112] See also Figure 7 , Figure 7 This is a structural block diagram of a model training device provided in one or more embodiments of this specification. Figure 7 As shown, the model training device 700 may include: a third construction module 701 and a training module 702.

[0113] The third construction module 701 is used to construct a frozen denoising network and an external transformer network; the frozen denoising network has the same architecture as the pre-trained text-video generation model, and is used to freeze the network weight parameters of the video denoising network in the pre-trained text-video generation model; the video denoising network includes a variational self-attention encoder for feature extraction and a preset number of diffusion transformer blocks; the external transformer network includes the same variational self-attention encoder as the video denoising network, and the preset number of illumination encoder blocks, and the illumination encoder block is a transformer architecture including a self-attention layer;

[0114] The training module 702 is used to train the pre-trained text video generation model, the external transformer network and the frozen denoising network based on the training samples and the objective function in the video data set to generate a trained text video generation model and an external transformer network; the objective function is a decoupling loss function of the predicted noise of the video denoising network and the predicted noise of the frozen denoising network; wherein, in any training iteration, the illumination feature extracted from each illumination encoder block is input into the corresponding diffusion transformer block, and the feature is fused with the video feature extracted from the diffusion transformer block and then input into the multi-head attention layer of the diffusion transformer block. The video data set is constructed based on the aforementioned video data set construction device 600.

[0115] In one possible implementation, the training module 702 is used to train the pre-trained text video generation model, the external transformer network and the frozen denoising network based on the training samples and the objective function in the video data set, including: the training module is used to: input the second video with text information annotated in the training samples in the video data set into the pre-trained text video generation model and the frozen denoising network respectively, and input the first video in the corresponding training sample into the external transformer network, and perform model training based on the objective function.

[0116] In a possible implementation, the training module 702 is used to perform feature fusion on the lighting feature and the video feature by element-by-element addition.

[0117] See also Figure 8 , Figure 8 This is a structural block diagram of a video generation device provided by one or more embodiments of this specification. Figure 8 As shown, the video generation device 800 may include: a selection module 801 and a second generation module 802.

[0118] A selection module 801 is used to select a target lighting track required for generating a target video from a plurality of lighting tracks in a 3D lighting grid;

[0119] The second generation module 802 is used to input the first video corresponding to the target lighting trajectory into the external transformer network, and input the target text information used to generate the target video into the text video generation model to generate the target video; wherein, in the process of generating the target video, the lighting features extracted from each lighting encoder block are input into the corresponding diffusion transformer block, and the features are fused with the video features extracted by the diffusion transformer block and then input into the multi-head attention layer of the diffusion transformer block. The external transformer network and the text video generation model are trained and generated by the aforementioned model training device 700.

[0120] See also Fig. 9 , Fig. 9 This is a structural block diagram of an electronic device provided in one or more embodiments of this specification. Fig. 9 As shown, the electronic device 900 may include a processor 901 and a memory 902; the memory 902 may be coupled to the processor 901. It is worth noting that the Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0121] In one possible implementation, the functions of the video data set construction device 600 , the model training device 700 , and the video generation device 800 may be integrated into the processor 901 .

[0122] In another possible implementation, the video dataset construction device 600, the model training device 700, and the video generation device 800 can be configured separately from the processor 901. For example, the video dataset construction device 600, the model training device 700, and the video generation device 800 can be configured as a chip connected to the processor 901, and corresponding processing can be implemented through the control of the processor 901.

[0123] In addition, in some optional implementations, the electronic device 900 may also include: a communication module, an input unit, an audio processor, a display, a power supply, etc. It is worth noting that the electronic device 900 does not necessarily include Fig. 9 In addition, the electronic device 900 may also include Fig. 9For components not shown, reference may be made to the prior art.

[0124] In some optional implementations, the processor 901 is sometimes also referred to as a controller or operation control, and may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device 900.

[0125] The memory 902 may be, for example, a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The memory 902 may store the information related to the video data set construction device 600, the model training device 700, and the video generation device 800, and may also store programs for executing the related information. The processor 901 may execute the program stored in the memory 902 to implement information storage or processing.

[0126] The input unit can provide input to the processor 901. The input unit is, for example, a key or a touch input device. The power supply can be used to provide power to the electronic device 900. The display can be used to display display objects such as images and text. The display can be, for example, an LCD display, but is not limited thereto.

[0127] The memory 902 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that stores information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 902 may also be some other type of device. The memory 902 includes a buffer memory (sometimes referred to as a buffer). The memory 902 may include an application / function storage unit for storing application programs and function programs or for executing a process for the operation of the electronic device 900 through the processor 901.

[0128] The memory 902 may also include a data storage unit for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit of the memory 902 may include various drivers for the communication function of the computer device and / or for executing other functions of the computer device (such as a messaging application, a contact book application, etc.).

[0129] The communication module is a transmitter / receiver that sends and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor 901 to provide input signals and receive output signals, which may be the same as in a conventional mobile communication terminal.

[0130] Based on different communication technologies, multiple communication modules may be provided in the same computer device, such as a cellular network module, a Bluetooth module and / or a wireless local area network module, etc. The communication module (transmitter / receiver) is also coupled to a speaker and a microphone via an audio processor to provide an audio output via the speaker and receive an audio input from the microphone, thereby realizing a common telecommunication function. The audio processor may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor is also coupled to the processor 901, so that recording can be performed on the machine through the microphone, and the sound stored on the machine can be played through the speaker.

[0131] One or more embodiments of the present specification also provide a computer-readable storage medium capable of implementing all the steps of the video dataset construction method, model training method, and video generation method in the above embodiments. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements all the steps of the video dataset construction method, model training method, and video generation method in the above embodiments.

[0132] Although one or more embodiments of this specification provide method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0133] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, devices (systems) or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0134] One or more embodiments of the present specification are described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to one or more embodiments of the present specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0137] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0138] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. For those of ordinary skill in the art, the specific meaning of the above terms in one or more embodiments of this specification can be understood according to the specific circumstances.

[0139] It should be noted that, in the absence of conflict, one or more embodiments of this specification and the features in the embodiments may be combined with each other. One or more embodiments of this specification are not limited to any single aspect, nor to any single embodiment, nor to any combination and / or replacement of these aspects and / or embodiments. Moreover, each aspect and / or embodiment of one or more embodiments of this specification may be used alone or in combination with one or more other aspects and / or embodiments thereof.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of one or more embodiments of this specification, rather than to limit them. Although one or more embodiments of this specification are described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of one or more embodiments of this specification, and they should all be included in the scope of one or more embodiments of this specification.

[0141] One or more embodiments of this specification are described above in conjunction with optional implementation methods, but these implementation methods are only exemplary and serve only as an illustration. On this basis, multiple replacements and improvements can be made to one or more embodiments of this specification, all of which fall within the scope of protection of one or more embodiments of this specification.

Claims

1. A method for constructing a video dataset, characterized in that: The method comprises: Constructing a plurality of lighting tracks in a 3D lighting grid; the 3D lighting grid is a three-dimensional grid whose grid points are used as lighting positions; the lighting track is a track connected by a plurality of the grid points; Acquire a first image set and at least one second image set corresponding to each lighting track; the first image set includes whiteboard images generated when point light sources are located at each lighting position of the corresponding lighting track and illuminate a whiteboard in a blank space; the second image set includes model images generated when the point light sources are located at each lighting position of the corresponding lighting track and illuminate a 3D model; the second image set corresponds to the 3D model one by one; Generate a first video corresponding to each first image set and a second video corresponding to each second image set; Annotating text information for each second video; Based on all the first videos and all the second videos with labeled text information, a video dataset for training a text video generation model is constructed; the video dataset includes multiple training samples, each training sample includes a first video corresponding to a lighting trajectory, and a second video with labeled text information of a 3D model corresponding to the lighting trajectory, and the lighting trajectory and the 3D model both correspond to the training samples one-to-one.

2. The method according to claim 1, characterized in that The first lighting track is any one of the multiple lighting tracks; Acquiring a first image set corresponding to a first lighting trajectory includes: Controlling the point light source to move sequentially to each lighting position of the first lighting track, after the point light source moves to any lighting position to illuminate the whiteboard in the blank space, generating a frame of whiteboard image through virtual engine rendering; All whiteboard images corresponding to the first lighting trajectory are constructed into a first image set corresponding to the first lighting trajectory in order of generation time.

3. The method according to claim 1, characterized in that The first lighting track is any one of the multiple lighting tracks; the third image set is any one of the at least one second image set corresponding to the first lighting track; The first 3D model is a 3D model corresponding to the third image set; Acquiring a third image set corresponding to the first lighting trajectory, including: Controlling the point light source to move to each lighting position of the first lighting track in sequence, and after the point light source moves to any lighting position to illuminate the first 3D model, generating a frame of model image through virtual engine rendering; All model images corresponding to the first lighting trajectory and the first 3D model are constructed into a third image set corresponding to the first lighting trajectory and the first 3D model in chronological order of generation time.

4. The method according to claim 1, characterized in that Annotate each second video with text information, including: Generate initial text information for each second video through an image language pre-training model; Performing title enhancement processing on the initial text information of each second video to generate enhanced text information of each second video; Corresponding enhanced text information is annotated for each second video.

5. The method according to any one of claims 1 to 4, characterized in that The 3D model is a 3D character model.

6. A model training method based on a video dataset, characterized in that: The video data set is constructed by the method described in any one of claims 1 to 5, the method comprising: Constructing a frozen denoising network and an external transformer network; the frozen denoising network has the same architecture as the pre-trained text-video generation model and is used to freeze the network weight parameters of the video denoising network in the pre-trained text-video generation model; the video denoising network includes a variational self-attention encoder for feature extraction and a preset number of diffusion transformer blocks; the external transformer network includes the same variational self-attention encoder as the video denoising network and the preset number of illumination encoder blocks, and the illumination encoder block is a transformer architecture including a self-attention layer; Based on the training samples and the objective function in the video data set, the pre-trained text video generation model, the external transformer network and the frozen denoising network are trained to generate a trained text video generation model and an external transformer network; the objective function is a decoupling loss function of the predicted noise of the video denoising network and the predicted noise of the frozen denoising network; Among them, in any training iteration process, the lighting features extracted from each lighting encoder block are input into the corresponding diffusion transformer block, and are fused with the video features extracted from the diffusion transformer block and then input into the multi-head attention layer of the diffusion transformer block.

7. The method according to claim 6, characterized in that Based on the training samples and the objective function in the video dataset, the pre-trained text video generation model, the external transformer network and the frozen denoising network are trained, including: The second video with labeled text information in the training sample in the video data set is input into the pre-trained text video generation model and the frozen denoising network respectively, and the first video in the corresponding training sample is input into the external transformer network, and the model training is performed based on the objective function.

8. The method according to claim 6, characterized in that The lighting feature and the video feature are fused by element-by-element addition.

9. A video generation method based on an external transformer network and a text-to-video generation model trained and generated by the method according to any one of claims 6 to 8, characterized in that: The method comprises: Selecting a target lighting track required to generate a target video from a plurality of lighting tracks in the 3D lighting grid; Inputting a first video corresponding to the target illumination trajectory into the external transformer network, and inputting target text information for generating a target video into the text video generation model to generate the target video; In the process of generating the target video, the lighting features extracted from each lighting encoder block are input into the corresponding diffusion transformer block, and are fused with the video features extracted by the diffusion transformer block and then input into the multi-head attention layer of the diffusion transformer block.

10. A video data set construction device, characterized in that: The device comprises: A first construction module is used to construct a plurality of lighting tracks in a 3D lighting grid; the 3D lighting grid is a three-dimensional grid whose grid points are used as lighting positions; the lighting track is a track connected by a plurality of the grid points; An acquisition module, used to acquire a first image set and at least one second image set corresponding to each lighting track; the first image set includes whiteboard images generated when point light sources are located at each lighting position of the corresponding lighting track and illuminate a whiteboard in a blank space; the second image set includes model images generated when the point light sources are located at each lighting position of the corresponding lighting track and illuminate a 3D model; the second image set corresponds to the 3D model one by one; A first generating module, used to generate a first video corresponding to each first image set and a second video corresponding to each second image set; A labeling module, used for labeling text information for each second video; The second construction module is used to construct a video data set for training a text video generation model based on all the first videos and all the second videos annotated with text information; the video data set includes multiple training samples, each training sample includes a first video corresponding to a lighting trajectory, and a second video annotated with text information of a 3D model corresponding to the lighting trajectory, and the lighting trajectory and the 3D model both correspond to the training samples one by one.

11. The device according to claim 10, characterized in that The first lighting track is any one of the multiple lighting tracks; The acquisition module is used to acquire a first image set corresponding to a first lighting trajectory, including: The acquisition module is used for: Controlling the point light source to move sequentially to each lighting position of the first lighting track, after the point light source moves to any lighting position to illuminate the whiteboard in the blank space, generating a frame of whiteboard image through virtual engine rendering; All whiteboard images corresponding to the first lighting trajectory are constructed into a first image set corresponding to the first lighting trajectory in order of generation time.

12. The device according to claim 10, characterized in that The first lighting track is any one of the multiple lighting tracks; the third image set is any one of the at least one second image set corresponding to the first lighting track; The first 3D model is a 3D model corresponding to the third image set; The acquisition module is used to acquire a third image set corresponding to the first lighting trajectory, including: The acquisition module is used for: Controlling the point light source to move to each lighting position of the first lighting track in sequence, and after the point light source moves to any lighting position to illuminate the first 3D model, generating a frame of model image through virtual engine rendering; All model images corresponding to the first lighting trajectory and the first 3D model are constructed into a third image set corresponding to the first lighting trajectory and the first 3D model in chronological order of generation time.

13. The device according to claim 10, characterized in that The annotation module is used to annotate text information for each second video, including: The annotation module is used for: Generate initial text information for each second video through an image language pre-training model; Performing title enhancement processing on the initial text information of each second video to generate enhanced text information of each second video; Corresponding enhanced text information is annotated for each second video.

14. The device according to any one of claims 10 to 13, characterized in that The 3D model is a 3D character model.

15. A model training device based on a video data set, characterized in that: The video data set is constructed by the video data set construction device according to any one of claims 10 to 14, and the model training device comprises: A third construction module is used to construct a frozen denoising network and an external transformer network; the frozen denoising network has the same architecture as the pre-trained text-video generation model, and is used to freeze the network weight parameters of the video denoising network in the pre-trained text-video generation model; the video denoising network includes a variational self-attention encoder for feature extraction and a preset number of diffusion transformer blocks; the external transformer network includes the same variational self-attention encoder as the video denoising network, and the preset number of illumination encoder blocks, and the illumination encoder block is a transformer architecture including a self-attention layer; A training module, for training the pre-trained text-video generation model, the external transformer network and the frozen denoising network based on the training samples and the objective function in the video data set, to generate a trained text-video generation model and an external transformer network; the objective function is a decoupling loss function of the predicted noise of the video denoising network and the predicted noise of the frozen denoising network; Among them, in any training iteration process, the lighting features extracted from each lighting encoder block are input into the corresponding diffusion transformer block, and are fused with the video features extracted from the diffusion transformer block and then input into the multi-head attention layer of the diffusion transformer block.

16. The model training device according to claim 15, characterized in that: The training module is used to train the pre-trained text video generation model, the external transformer network and the frozen denoising network based on the training samples and the objective function in the video data set, including: The training module is used to: The second video with labeled text information in the training sample in the video data set is input into the pre-trained text video generation model and the frozen denoising network respectively, and the first video in the corresponding training sample is input into the external transformer network, and the model training is performed based on the objective function.

17. The model training device according to claim 15, characterized in that: The training module is used to perform feature fusion on the lighting feature and the video feature by element-by-element addition.

18. A video generation device based on an external transformer network and a text-to-video generation model trained and generated by the model training device according to any one of claims 15 to 17, characterized in that: The video generating device comprises: A selection module, used for selecting a target lighting track required for generating a target video from a plurality of lighting tracks in the 3D lighting grid; A second generation module is used to input the first video corresponding to the target lighting trajectory into the external transformer network, and input the target text information used to generate the target video into the text video generation model to generate the target video; In the process of generating the target video, the lighting features extracted from each lighting encoder block are input into the corresponding diffusion transformer block, and are fused with the video features extracted by the diffusion transformer block and then input into the multi-head attention layer of the diffusion transformer block.

19. An electronic device, characterized in that: The electronic device comprises: A memory for storing a computer program product; A processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the method described in any one of claims 1-5, 6-8, and 9 is implemented.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when executed, implement the method described in any one of claims 1-5, 6-8, and 9 above.

Images