Diffusion model construction method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a diffusion model construction method and device, equipment and a storage medium. The method comprises the steps that a pre-training model is acquired, a target model structure of a target diffusion model is constructed, and the target model structure comprises a basic network and a target up-sampling stack network; performing initialization operation on the target model structure through the pre-training model to obtain an initial diffusion model; sequentially carrying out time sequence modeling processing and text embedding processing on the initial diffusion model to obtain a processed initial diffusion model; and training the modeled initial diffusion model according to a training set to obtain the target diffusion model, the training set comprising a plurality of pieces of text information, and the target diffusion model being used for analyzing the plurality of pieces of text information to obtain corresponding video information. The method aims at generating the diffusion model which is excellent in performance and universal, reduces the construction cost of the diffusion model, and can meet the requirements of scenes such as the financial field.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to a method for constructing a diffusion model, an apparatus for constructing a diffusion model, a computer device, and a computer-readable storage medium. Background Art

[0002] In recent years, image generation technology based on diffusion models has made significant progress, especially the success of text-to-image generation, but it still faces challenges in generating realistic and temporally coherent animation frame sequences. Diffusion models not only need to learn image generation for each frame, but also need to capture the temporal dynamics between frames, which is costly.

[0003] In addition, the construction of existing diffusion models usually relies on data sets in specific fields and has poor flexibility. For example, Ping An Securities in the financial field needs to generate video content related to the financial market. Therefore, it is necessary to build a universal diffusion model to improve quality and diversity, especially to meet the needs of various scenarios such as the financial field. Summary of the invention

[0004] The present application provides a method for constructing a diffusion model, an apparatus for constructing a diffusion model, a computer device and a computer-readable storage medium, aiming to generate a diffusion model with better performance and versatility, and reduce the construction cost of the diffusion model.

[0005] To achieve the above purpose, the present application also provides a method for constructing a diffusion model, comprising:

[0006] Obtain a pre-trained model and construct a target model structure of a target diffusion model, wherein the target model structure includes a base network and a target upsampling stack network;

[0007] Initializing the target model structure through the pre-trained model to obtain an initial diffusion model;

[0008] Performing sequential modeling and text embedding processing on the initial diffusion model to obtain a processed initial diffusion model;

[0009] The modeled initial diffusion model is trained according to a training set to obtain the target diffusion model, wherein the training set includes a number of text information, and the target diffusion model is used to analyze the number of text information to obtain corresponding video information.

[0010] To achieve the above purpose, the present application also provides a device for constructing a diffusion model, comprising:

[0011] An acquisition module, used to acquire a pre-trained model and construct a target model structure of a target diffusion model, wherein the target model structure includes a base network and a target upsampling stack network;

[0012] An initialization module, used to initialize the target model structure through the pre-trained model to obtain an initial diffusion model;

[0013] A processing module, used for sequentially performing time series modeling processing and text embedding processing on the initial diffusion model to obtain a processed initial diffusion model;

[0014] The training module is used to train the modeled initial diffusion model according to the training set to obtain the target diffusion model, wherein the training set includes a number of text information, and the target diffusion model is used to analyze the several text information to obtain corresponding video information.

[0015] In addition, to achieve the above-mentioned purpose, the present application also provides a computer device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the steps of the method for constructing a diffusion model provided in any one of the embodiments of the present application when executing the computer program.

[0016] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the steps of the method for constructing a diffusion model described in any one of the embodiments of the present application.

[0017] The present application discloses a method for constructing a diffusion model, a device for constructing a diffusion model, a computer device, and a computer-readable storage medium. The method includes obtaining a pre-trained model and constructing a target model structure of a target diffusion model, wherein the target model structure includes a basic network and a target upsampling stack network. Further, the target model structure can be initialized by the pre-trained model to obtain an initial diffusion model, and the initial diffusion model is sequentially subjected to a time series modeling process and a text embedding process to obtain a processed initial diffusion model. Finally, the modeled initial diffusion model can be trained according to a training set to obtain a target diffusion model, wherein the training set includes a number of text information, and the target diffusion model is used to analyze the text information to obtain corresponding video information. The present application reduces the complexity of training the model from scratch by using a pre-trained model to initialize the target model structure, significantly shortens the training time, and improves the generalization ability and convergence speed of the model. Further, the basic network and the target upsampling stack network are introduced into the target model structure, and the complex relationship between text information and video generation can be effectively captured in combination with the time series modeling process and the text embedding process, thereby improving the accuracy and consistency of the generated video. In addition, after the diffusion model is trained with the training set, it can efficiently parse text information and generate video information related to the text semantics, realizing the multimodal conversion from text to video and expanding the application scope of the model. In summary, the embodiment of the present application can generate a diffusion model with better performance and versatility, and reduce the construction cost of the diffusion model, especially to meet the needs of various scenarios such as the financial field. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a scene schematic diagram of a method for constructing a diffusion model provided in an embodiment of the present application;

[0020] Figure 2 It is a flow chart of a method for constructing a diffusion model provided in an embodiment of the present application;

[0021] Figure 3 is a schematic block diagram of a device for constructing a diffusion model provided in an embodiment of the present application;

[0022] Figure 4 It is a schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions. In addition, although the functional modules are divided in the device schematic, in some cases, the module division can be different from that in the device schematic.

[0025] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0026] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0027] like Figure 1 As shown, the method for constructing a diffusion model provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. The application environment includes a terminal device 110 and a server 120, wherein the terminal device 110 can communicate with the server 120 through a network. Specifically, the server 120 can obtain a pre-trained model and construct a target model structure of a target diffusion model, wherein the target model structure includes a basic network and a target upsampling stack network; the target model structure is initialized by the pre-trained model to obtain an initial diffusion model; the initial diffusion model is sequentially subjected to time series modeling processing and text embedding processing to obtain a processed initial diffusion model; the modeled initial diffusion model is trained according to a training set to obtain the target diffusion model, wherein the training set includes a number of text information, and the target diffusion model is used to analyze the several text information to obtain corresponding video information, thereby sending the target diffusion model to the terminal device 110. Among them, the server 120 can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal device 110 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0028] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of a method for constructing a diffusion model provided in an embodiment of the present application. Figure 2 As shown, the method for constructing the diffusion model can be implemented through steps S11 to S14.

[0029] Step S11: Obtain a pre-trained model and construct a target model structure of a target diffusion model.

[0030] Among them, the target model structure includes a basic network and a target upsampling stack network.

[0031] Optionally, the output of the base network includes a video with a preset number of frames; the target upsampling stack network includes a first upsampling stack network, a second upsampling stack network and a third upsampling stack network, the output video frame number of the first upsampling stack network is the same as the output video frame number of the base network; the output video frame number of the second upsampling stack network is greater than the output video frame number of the first upsampling stack network; the output of the third upsampling stack network is greater than the output video frame number of the second upsampling stack network.

[0032] By way of example, the preset frame numbers include 76×32×64, 76×32×32, and 76×64×64, etc. The present application takes the preset frame number of 76×64×64 as an example for explanation, that is, the output of the basic network includes 76×64×64 frames of video; the target upsampling stack network includes a first upsampling stack network, a second upsampling stack network, and a third upsampling stack network, and the output of the first upsampling stack network includes 76×64×64 frames of video; the output of the second upsampling stack network is greater than the output video frame number of the first upsampling stack network; the output of the third upsampling stack network is greater than the output video frame number of the second upsampling stack network, for example, the output of the second upsampling stack network includes 76×256×256 frames of video; the output of the second upsampling stack network includes 76×1024×1024 frames of video, and the present application does not limit this.

[0033] For example, the base network is one of the core parts of the target model structure, which can initially generate a low-resolution version of the video frame. The output of the base network is a video of size 76×64×64 frames, that is, the temporal dimension of the video has 76 frames, and the spatial resolution of each frame is 64×64 pixels, which can provide the initial input for the subsequent upsampling network.

[0034] For example, the upsampling stack network is used to gradually upscale the low-resolution video output by the base network to a higher resolution. The network consists of multiple upsampling modules, each of which is responsible for upscaling different resolutions to ensure that the video quality is gradually optimized. The first upsampling stack network receives the 76×64×64 video output by the base network and performs the first upsampling. This process improves the details and resolution of the video, introduces richer contextual information and detail modification, and improves the video quality.

[0035] For example, the second upsampling stack network receives the output of the first upsampling stack network and increases the resolution to 76×256×256. Thus, the number of pixels per frame of the video can be greatly increased, so that the generated video frame has finer image details, which is closer to the requirements in practical applications.

[0036] For example, the third upsampling stack network continues to improve the resolution based on the second upsampling stack network, and finally increases the resolution of the video to 76×1024×1024, which meets the high-definition standard and can ensure that the generated video has high-quality visual effects and is suitable for application requirements of various high-resolution scenarios.

[0037] For example, combining the base network and the target upsampling stack network, the type of the target model can be different according to the application scenario and model design, including the following: Generative adversarial network combined with diffusion model: Generative adversarial network can be used as the basic part of the diffusion model, the diffusion model is responsible for generating the preliminary structure of the image, and the discriminator part of the generative adversarial network can enhance the realism of the image; Variational autoencoder combined with diffusion model, variational autoencoder can be used as the base network for learning the potential representation of the image, and then the diffusion model is used for generation and detail enhancement. Pure diffusion model, the pure diffusion model directly generates high-quality images from noise, relying on multi-step upsampling stack networks to gradually improve the resolution and quality of the image.

[0038] The above embodiments can construct the target model structure of the target diffusion model, and realize the conversion of the initial low-resolution video into a high-definition long-time sequence video using an upsampling stack network. Each upsampling stack network performs detail enhancement at different resolutions to ensure that the final generated video meets visual and business requirements.

[0039] Step S12: Initialize the target model structure through the pre-trained model to obtain an initial diffusion model.

[0040] Among them, the pre-trained model can be a model that has performed well in related tasks (such as image or video generation, feature extraction, etc.), such as a diffusion model, a variational autoencoder, a generative adversarial network, etc., and this application does not limit this.

[0041] For example, the pre-trained model can use its existing weights and parameters to initialize the target diffusion model, thereby saving time for subsequent training and providing a good initial state so that the model can converge faster.

[0042] For example, the weights of the pre-trained model can be used to initialize the base network and the target upsampling stack network. By introducing the pre-trained weights into the target model structure, it can ensure that the model has reasonable parameter settings at an early stage to better adapt to the specific text-to-video generation task.

[0043] Step S13: sequentially perform time series modeling processing and text embedding processing on the initial diffusion model to obtain a processed initial diffusion model.

[0044] For example, by introducing a temporal modeling module (such as a recurrent neural network, a long short-term memory network, etc.) to model the temporal relationship of video frames, the dependencies between frames can be effectively learned, helping the model to maintain the coherence of frames when generating new videos and avoid flickering or frame skipping. After the temporal modeling process, the initial diffusion model can generate video sequences with continuous temporal dynamics, rather than just individual static frames.

[0045] For example, the initial diffusion model can also be processed by time coding operation for time series modeling. Among them, time coding is usually used to embed time step information (such as the number of time steps in the diffusion process) into the model so that the model can distinguish the states under different time steps. For example, the time step can be encoded by a sine and cosine function to capture the periodic relationship between the time steps. Furthermore, the encoded time step information can be used as a vector and added or concatenated with the features in the model, thereby introducing time series information in the diffusion process.

[0046] The initial diffusion model is processed by time series modeling through time encoding operation, which can effectively capture the noise distribution characteristics under different time steps and has low requirements on computing resources.

[0047] For example, text embedding technology can convert text into a high-dimensional vector representation to capture the semantic information therein, such as word vectors (such as Word2Vec, GloVe), sentence vector models (such as BERT, GPT, etc.), which are not limited in this application.

[0048] For example, a pre-trained word embedding model (such as Word2Vec, GloVe, BERT, etc.) can be used to convert text into a fixed-length vector representation. For example, the input text can be segmented, a vocabulary index can be generated, and each word can be mapped to a corresponding word vector using a pre-trained word embedding model, so that the word vectors of the entire sentence can be averaged or weighted averaged to obtain the overall embedding vector of the text.

[0049] For example, the text embedding vector can be closely integrated with the video generation module, and through the attention mechanism or cross-modal transformation module, the key information in the text can be mapped to the video generation process, thereby achieving efficient conversion from text to video. Furthermore, through text embedding processing, the initial diffusion model can generate corresponding video clips based on the given text information, realizing the modal conversion from text to video.

[0050] In the above embodiments, the target model can obtain initial parameter settings through pre-training model initialization. Furthermore, the temporal consistency and semantic expression ability of the model can be further enhanced through temporal modeling processing and text embedding processing, so that the final processed initial diffusion model can generate dynamic, coherent and semantically relevant video content based on text information.

[0051] Step S14: training the initial diffusion model after modeling according to the training set to obtain a target diffusion model.

[0052] The training set includes some text information, and the target diffusion model is used to analyze the some text information to obtain corresponding video information.

[0053] For example, the initial diffusion model can be adjusted based on the text and video data in the training set to learn the mapping relationship from text to video, and after training and optimization, a target diffusion model is obtained to generate a corresponding video based on the input text information.

[0054] The above embodiments enable the model to have the ability to generate high-quality, time-consistent videos.

[0055] The method for constructing a diffusion model disclosed in the embodiment of the present application includes obtaining a pre-trained model and constructing a target model structure of a target diffusion model, wherein the target model structure includes a basic network and a target upsampling stack network. Further, the target model structure can be initialized by the pre-trained model to obtain an initial diffusion model, and the initial diffusion model is sequentially subjected to a time series modeling process and a text embedding process to obtain a processed initial diffusion model. Finally, the modeled initial diffusion model can be trained according to a training set to obtain a target diffusion model, wherein the training set includes a number of text information, and the target diffusion model is used to analyze the text information to obtain corresponding video information. The present application reduces the complexity of training the model from scratch by using a pre-trained model to initialize the target model structure, significantly shortens the training time, and improves the generalization ability and convergence speed of the model. Further, the basic network and the target upsampling stack network are introduced into the target model structure, and the complex relationship between text information and video generation can be effectively captured in combination with the time series modeling process and the text embedding process, thereby improving the accuracy and consistency of the generated video. In addition, after the diffusion model is trained with the training set, it can efficiently parse text information and generate video information related to the text semantics, realizing the multimodal conversion from text to video and expanding the application scope of the model. In summary, the embodiment of the present application can generate a diffusion model with better performance and versatility, and reduce the construction cost of the diffusion model.

[0056] Optionally, the initial diffusion model is subjected to time series modeling processing, including: adding a dimension operation to the time axis in the initial diffusion model so that the dimension of the time axis is a 3D convolution dimension; and / or, adding a time attention layer operation to the initial diffusion model so that the initial diffusion model includes a time attention mechanism.

[0057] For example, by processing the video frame sequence in the initial diffusion model, a dimension is added to the time axis, that is, the video data is expanded from the 2D convolution dimension to the 3D convolution dimension, so that the model can extract features in both the time and space dimensions, thereby enhancing the model's perception ability in the time dimension.

[0058] For example, the temporal attention mechanism is based on the self-attention mechanism, which determines which parts should be paid more attention by calculating the dependencies between different positions. The temporal attention layer can assign different weights to the frames of each time step, that is, dynamically adjust the attention weight according to the temporal relationship between each frame and other frames, and highlight the frames or moments that are more relevant to the overall video generation.

[0059] For example, the temporal attention mechanism can calculate the relevance of video frames at different time points (usually using dot products or other similarity metrics) and assign weights based on their relationships with other frames. This can help the model focus on moments that are critical to scene or action changes during video generation, while reducing attention to irrelevant or minor moments.

[0060] The above embodiments can significantly enhance the model's perception of temporal information by performing temporal modeling on the initial diffusion model, thereby ensuring that the initial diffusion model performs well in the video generation task and can generate high-quality, temporally consistent video content based on text.

[0061] Optionally, the initial diffusion model after modeling is trained according to the training set to obtain a target diffusion model, including: obtaining the training set, and annotating the training set to obtain corresponding annotation results, wherein the annotation results include image information and video information corresponding to each text information; training the initial diffusion model after modeling through the training set and the annotation results to obtain the target diffusion model.

[0062] Specifically, the annotation results corresponding to the training set can be used as the label of the group of input data, and then each group of labeled training sets can be input into the initial diffusion model for supervised learning. When the training end conditions are met, such as the number of training times reaches the number threshold or the output accuracy of the model reaches the accuracy threshold, the training is terminated to obtain the trained target diffusion model.

[0063] In the embodiment of the present application, the training set and the annotation results can be input into the initial diffusion model for supervised learning, and then the target diffusion model can be trained to obtain the target diffusion model. Thus, the video content corresponding to the text information can be output based on the target diffusion model.

[0064] On the basis of the above embodiment, the initial diffusion model after modeling is trained through the training set and the annotation results to obtain the target diffusion model, including: splicing the image information and the video information on the time axis to obtain the spliced ​​annotation results; training the initial diffusion model after modeling based on the training set and the spliced ​​annotation results to obtain the target diffusion model.

[0065] It should be understood that by splicing image information and video information on the timeline, the changes and characteristics of the target object at different times can be more comprehensively reflected, increasing the diversity of data and helping to expand the generalization ability of the model. Furthermore, the spliced ​​annotation results combine information from multiple time points, which can reduce the error of data annotation at a single moment, improve the accuracy of annotation, and thus improve the training effect of the model. It can also alleviate the noise interference that may exist in a single image or video frame to a certain extent, and improve the adaptability of the model to complex scenes.

[0066] The above embodiments can enhance data quality and diversity during the diffusion model training process, and improve the generalization ability and practical application effect of the model.

[0067] Optionally, after obtaining the target diffusion model, it also includes: iteratively training the target diffusion model to extract data features and calculate the loss function; iteratively training the loss function using a preset method for the purpose of reducing the value of the loss function until the expected threshold requirement is met; based on the iteratively trained loss function, obtaining the iterative target diffusion model.

[0068] It can be understood that in order to train a target diffusion model with higher accuracy, the target diffusion model can be repeatedly trained to continuously reduce the loss function until the loss function meets the expected threshold requirement, and then a more accurate video content corresponding to the text input can be obtained based on the iterated target diffusion model.

[0069] It should be noted that the present application does not limit the above-mentioned preset method and expected threshold. For example, the preset method may be a gradient descent algorithm, a batch gradient descent algorithm, a stochastic gradient descent algorithm, etc. The present application takes the gradient descent algorithm as an example for explanation.

[0070] The purpose of the gradient descent algorithm is to find the minimum value of the loss function or converge to the minimum value through iteration. In a geometric sense, the gradient descent algorithm means that the gradient decreases fastest in the direction opposite to the vector where the function changes the fastest, so it is easier to find the minimum value of the function. Based on this, in an embodiment of the present application, the target diffusion model can be repeatedly iterated and trained by the gradient descent algorithm so that the loss function is continuously reduced, thereby reducing the error of the calculation result.

[0071] In the embodiment of the present application, the denoising model is repeatedly trained by using a gradient descent algorithm so that the loss function is continuously reduced to obtain an iterative target diffusion model, and then a more accurate video content corresponding to the text input can be obtained based on the iterative target diffusion model.

[0072] Optionally, after the modeled initial diffusion model is trained according to the training set to obtain the target diffusion model, it also includes: deploying the target diffusion model to a preset platform so that the user can input a text description based on the preset platform and obtain a video corresponding to the text description.

[0073] For example, the trained target diffusion model can be deployed on a preset platform (such as a cloud platform, server or mobile device), which can provide users with an interface where they can enter text descriptions, and the platform processes these text inputs through the target diffusion model.

[0074] For example, the user enters a text description on the preset platform, and the text description will be converted into a feature vector that the model can understand, and processed by the target diffusion model. Among them, the target diffusion model can start to generate the corresponding video based on the input text description, and use the correspondence between text and video learned in training to gradually generate clear video clips from the initial noisy video frame until a video that highly matches the text description is obtained.

[0075] For example, a training set related to the financial field can be obtained to optimize the initial diffusion model, so that the model can generate corresponding visual content when a text description of a financial scenario is input. The training set may include but is not limited to the following: Financial data visualization, such as chart videos of data such as stock market trends, portfolio performance, and interest rate fluctuations. Financial product promotion, such as promotional videos for financial products such as insurance, loans, and credit cards, generating animations or video clips with financial elements based on the description of the product. Financial education videos, such as visualization videos explaining economic concepts (such as inflation, equity financing, and debt management).

[0076] For example, after training, the initial diffusion model can understand and generate video content related to financial scenarios, thereby obtaining a target diffusion model. The target diffusion model can generate a video that is highly consistent with the input financial text description. For example, when the input text description predicts the inflation trend in the next year, the target diffusion model can generate a dynamic visualization video showing the inflation curve, currency and commodity price fluctuations. For example, the target diffusion model can be deployed to a preset platform, which can be a portal or mobile application of a financial service provider (such as a bank, an insurance company, or a fintech company). Users can enter text descriptions through the preset platform platform and generate video content related to it. For example, users can enter text descriptions through an input box or a voice recognition function, and the descriptions can include financial consulting, product introductions, market analysis, and other content. Further, the preset platform can start the model to reason and generate videos based on the text descriptions entered by the user through integration with the target diffusion model. For example, a bank can generate a video to promote a financial product, and an investment advisor can generate a short video of market analysis and share it with customers, which is not limited in this application.

[0077] The above method can successfully apply the target diffusion model to the task of text description to video generation, and by deploying it on the platform, users can easily use this technology to generate high-quality video content.

[0078] See also Figure 3 , Figure 3 This is a schematic block diagram of a diffusion model construction device provided in an embodiment of the present application. The diffusion model construction device can be configured in a server to execute the aforementioned diffusion model construction method.

[0079] like Figure 4 As shown, the diffusion model construction device 200 includes: an acquisition module 201, an initialization module 202, a processing module 203 and a training module 204.

[0080] An acquisition module 201 is used to acquire a pre-trained model and construct a target model structure of a target diffusion model, wherein the target model structure includes a base network and a target upsampling stack network;

[0081] An initialization module 202 is used to initialize the target model structure through the pre-trained model to obtain an initial diffusion model;

[0082] A processing module 203 is used to sequentially perform time series modeling processing and text embedding processing on the initial diffusion model to obtain a processed initial diffusion model;

[0083] The training module 204 is used to train the modeled initial diffusion model according to the training set to obtain the target diffusion model, wherein the training set includes a number of text information, and the target diffusion model is used to analyze the several text information to obtain corresponding video information.

[0084] The initialization module 202 is further configured to perform a dimension adding operation on the time axis in the initial diffusion model so that the dimension of the time axis is a 3D convolution dimension; and / or,

[0085] An operation of adding a temporal attention layer is performed in the initial diffusion model so that the initial diffusion model includes a temporal attention mechanism.

[0086] The training module 204 is further used to obtain the training set, and annotate the training set to obtain corresponding annotation results, wherein the annotation results include image information and video information corresponding to each of the text information;

[0087] The modeled initial diffusion model is trained using the training set and the annotation results to obtain the target diffusion model.

[0088] The training module 204 is further used to splice the image information and the video information on the time axis to obtain a spliced ​​annotation result; and to train the modeled initial diffusion model based on the training set and the spliced ​​annotation result to obtain the target diffusion model.

[0089] The training module 204 is also used to iteratively train the target diffusion model to extract data features and calculate the loss function; iteratively train the loss function using a preset method for the purpose of reducing the value of the loss function until the expected threshold requirement is met; based on the iteratively trained loss function, obtain the iterative target diffusion model.

[0090] The training module 204 is further used to deploy the target diffusion model to a preset platform, so that a user can input a text description based on the preset platform and obtain a video corresponding to the text description.

[0091] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0092] The method and apparatus of the present application can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer terminal devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices.

[0093] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in the form of a computer program. Figure 4 Runs on the computer device shown.

[0094] See also Figure 4 , Figure 4 1 is a schematic diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0095] like Figure 4 As shown, the computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a volatile storage medium, a non-volatile storage medium and an internal memory.

[0096] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any method for constructing a diffusion model.

[0097] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0098] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any method for constructing a diffusion model.

[0099] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that the structure of the computer device is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0100] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0101] Among them, in some embodiments, the processor is used to obtain a pre-trained model and construct a target model structure of a target diffusion model, wherein the target model structure includes a basic network and a target upsampling stack network; the target model structure is initialized by the pre-trained model to obtain an initial diffusion model; the initial diffusion model is sequentially subjected to time series modeling processing and text embedding processing to obtain a processed initial diffusion model; the modeled initial diffusion model is trained according to a training set to obtain the target diffusion model, wherein the training set includes a number of text information, and the target diffusion model is used to analyze the a number of text information to obtain corresponding video information.

[0102] In some embodiments, the processor is also used to add a dimension operation to the time axis in the initial diffusion model so that the dimension of the time axis is a 3D convolution dimension; and / or, to add a temporal attention layer operation to the initial diffusion model so that the initial diffusion model includes a temporal attention mechanism.

[0103] In some embodiments, the processor is also used to obtain the training set and annotate the training set to obtain corresponding annotation results, wherein the annotation results include image information and video information corresponding to each of the text information; the initial diffusion model after modeling is trained through the training set and the annotation results to obtain the target diffusion model.

[0104] In some embodiments, the processor is further used to splice the image information and the video information on the timeline to obtain a spliced ​​annotation result; and train the modeled initial diffusion model based on the training set and the spliced ​​annotation result to obtain the target diffusion model.

[0105] In some embodiments, the processor is also used to iteratively train the target diffusion model to extract data features and calculate a loss function; iteratively train the loss function using a preset method for the purpose of reducing the value of the loss function until an expected threshold requirement is met; and based on the iteratively trained loss function, obtain the iterative target diffusion model.

[0106] In some implementations, the processor is further configured to deploy the target diffusion model to a preset platform, so that a user can input a text description based on the preset platform and obtain a video corresponding to the text description.

[0107] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed, any method for constructing a diffusion model provided in the embodiment of the present application is implemented.

[0108] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0109] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like.

[0110] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for constructing a diffusion model, characterized in that: The method comprises: Obtain a pre-trained model and construct a target model structure of a target diffusion model, wherein the target model structure includes a base network and a target upsampling stack network; Initializing the target model structure through the pre-trained model to obtain an initial diffusion model; Performing sequential modeling and text embedding processing on the initial diffusion model to obtain a processed initial diffusion model; The modeled initial diffusion model is trained according to a training set to obtain the target diffusion model, wherein the training set includes a number of text information, and the target diffusion model is used to analyze the several text information to obtain corresponding video information.

2. The method according to claim 1, characterized in that The performing time series modeling processing on the initial diffusion model includes: Adding a dimension to the time axis in the initial diffusion model so that the dimension of the time axis is a 3D convolution dimension; and / or, An operation of adding a temporal attention layer is performed in the initial diffusion model so that the initial diffusion model includes a temporal attention mechanism.

3. The method according to claim 1, characterized in that The step of training the modeled initial diffusion model according to the training set to obtain the target diffusion model includes: Acquire the training set, and annotate the training set to obtain corresponding annotation results, wherein the annotation results include image information and video information corresponding to each of the text information; The modeled initial diffusion model is trained using the training set and the annotation results to obtain the target diffusion model.

4. The method according to claim 3, characterized in that: The training of the modeled initial diffusion model using the training set and the annotation result to obtain the target diffusion model includes: Splicing the image information and the video information on a time axis to obtain a spliced ​​annotation result; The modeled initial diffusion model is trained based on the training set and the spliced ​​annotation results to obtain the target diffusion model.

5. The method according to claim 3, characterized in that: After obtaining the target diffusion model, the method further includes: Iteratively training the target diffusion model to extract data features, and calculating a loss function; Iteratively training the loss function using a preset method for the purpose of reducing the value of the loss function until the expected threshold value is met; Based on the loss function after iterative training, the target diffusion model after iteration is obtained.

6. The method according to claim 1, characterized in that The output of the basic network includes a video with a preset number of frames; the target upsampling stack network includes a first upsampling stack network, a second upsampling stack network and a third upsampling stack network, the output video frame number of the first upsampling stack network is the same as the output video frame number of the basic network; the output video frame number of the second upsampling stack network is greater than the output video frame number of the first upsampling stack network; the output of the third upsampling stack network is greater than the output video frame number of the second upsampling stack network.

7. The method according to claim 1, characterized in that After the initial diffusion model is trained according to the training set to obtain the target diffusion model, the method further includes: The target diffusion model is deployed to a preset platform, so that a user can input a text description based on the preset platform and obtain a video corresponding to the text description.

8. A device for constructing a diffusion model, characterized in that: The construction device comprises: An acquisition module, used to acquire a pre-trained model and construct a target model structure of a target diffusion model, wherein the target model structure includes a base network and a target upsampling stack network; An initialization module, used to initialize the target model structure through the pre-trained model to obtain an initial diffusion model; A processing module, used for sequentially performing time series modeling processing and text embedding processing on the initial diffusion model to obtain a processed initial diffusion model; The training module is used to train the modeled initial diffusion model according to the training set to obtain the target diffusion model, wherein the training set includes a number of text information, and the target diffusion model is used to analyze the several text information to obtain corresponding video information.

9. A computer device, characterized in that: include: A memory and a processor; wherein the memory is connected to the processor and is used to store a program; the processor is used to implement the steps of the method for constructing a diffusion model as described in any one of claims 1 to 7 by running the program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the steps of the method for constructing a diffusion model according to any one of claims 1 to 7.

Images