Digital media work automatic generation method based on large language model

Abstract

The invention discloses a large language model-based digital media work automatic generation method, which comprises the following steps of: S1, receiving text description input by a user, and analyzing text information by utilizing a DeepSeek large language model; s2, determining a target mode and selecting a corresponding generation model; s3, constructing and generating input; s4, the generated input is transmitted into an improved VQ-VAE-2 model, and the model is fused with a long and short term memory network and a cyclic auto-encoder to process time sequence data; s5, dynamically adjusting potential space distribution through an adaptive discrete space and variational reasoning method; s6, performing semantic consistency check on the generated content by using a DeepSeek model; s7, optimizing and generating output according to a consistency result; and S8, outputting the image, video or audio works. According to the invention, language-driven multi-modal content generation and semantic control are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of content generation, and particularly relates to a digital media work automatic generation method based on a large language model. BACKGROUND

[0002] With the rapid development of deep learning technology, content generation models based on neural networks are widely used in the generation tasks of various digital media such as images, videos, audios and texts. In particular, in the field of natural language processing, the modeling capability of large language models is continuously enhanced, making text-driven multi-modal generation one of the hot research directions in artificial intelligence technology. Existing large language models have shown excellent performance in text generation, text summarization, dialogue generation and other tasks, and have also begun to try to integrate with image, audio and other modalities in multi-modal generation tasks to realize cross-modal generation from language to media.

[0003] In terms of generation of visual content such as images and videos, the variational autoencoder (VAE), the generative adversarial network (GAN) and the vector quantization variational autoencoder (VQ-VAE) model proposed in recent years have their own advantages in generation quality, expression capability and controllability of latent space. Among them, VQ-VAE-2 has achieved good results in high-fidelity image generation by introducing a hierarchical structure and a discrete latent space, and has become one of the mainstream generation frameworks. In order to enhance the language understanding capability of the model, some researches begin to embed the language model into the image generation process to guide the generation of content through text description. However, the existing methods are mostly based on static model structure, lack the modeling capability of complex time series data (such as video and speech), and have deficiencies in the expression and regulation of latent space structure, making it difficult to adapt to diversified and dynamic generation requirements.

[0004] In addition, the current large language model mainly undertakes the functions of language understanding and content planning in multi-modal generation tasks, and has not yet formed a stable mechanism for deep integration of structure and generation model. There are still great challenges in semantic control, style consistency, time continuity and other aspects. The existing technology has not solved the problem of how to accurately map language semantics to high-quality multi-modal output. The generation results often lack coherence and consistency, and it is difficult to carry out flexible content customization and latent space control.

[0005] In summary, the combination of existing large language models and generation models still has the technical problems of single modeling method, lack of time modeling capability, unadjustable latent space, and insufficient matching degree between generated content and semantics in multi-modal automatic generation tasks.

[0006] Therefore, how to provide a digital media work automatic generation method based on a large language model is a problem that those skilled in the art need to solve. SUMMARY

[0007] An object of the present application is to provide an automatic generation method of digital media works based on large language models. The present application combines the semantic understanding ability of large language models and the content construction ability of generation models. An improved VQ-VAE-2 model is used as a generation skeleton. A long short-term memory network and a recurrent autoencoder are combined to model the structure of time series data such as images, videos, or audio. An adaptive discrete space adjustment mechanism and a variational inference method are introduced to dynamically optimize the latent space, improving the flexibility and accuracy of the generated expression. The present application describes the whole process from text input analysis, modal generation model selection, latent space coding optimization, to semantic consistency verification. It has the advantages of wide cross-modal support, strong controllability of generated structure, and high precision of semantic alignment.

[0008] According to an embodiment of the present application, an automatic generation method of digital media works based on large language models comprises the following steps:

[0009] S1, receiving a text description input by a user, using a DeepSeek large language model to analyze the text description and extract creative style, content requirements, and target modalities;

[0010] S2, determining the target modalities according to the extracted creative information, and selecting a generation model corresponding to the target modalities;

[0011] S3, combining the generation conditions with the key information in the text description according to the selected generation model to construct a corresponding generation input;

[0012] S4, inputting the generation input into an improved VQ-VAE-2 model. The improved VQ-VAE-2 model combines a long short-term memory network module and a recurrent autoencoder to handle the time dependence in time series data, generating image, video, or audio content corresponding to the target modalities;

[0013] S5, in the improved VQ-VAE-2 model, using an adaptive discrete space and a variational inference method to dynamically adjust the distribution of the latent space according to the complexity of the data, optimizing the generated content;

[0014] S6, using a DeepSeek large language model to perform semantic consistency checking on the generated image, video, or audio content;

[0015] S7, optimizing and adjusting the generated content according to the results of the semantic consistency checking;

[0016] S8, outputting the generated digital media works, including image, video, or audio files, and returning them to the user.

[0017] Optionally, the S2 specifically comprises:

[0018] S21, receiving the creation information extracted from the S1 step, including the creation style, content requirements and target modalities;

[0019] S22, determining the target modalities according to the creation information, including images, videos and audios;

[0020] S23, if the target modality is an image or a video, selecting a generation model related to image or video generation for generating image or video content;

[0021] S24, if the target modality is an audio, selecting a generation model related to audio generation for generating audio content;

[0022] S25, constructing a generation input according to the selected generation model and the target modality, including the creation style, content details and key information of the generation target;

[0023] S26, adjusting the parameters of the generation input according to the creation style and the content requirements, and the adjustment basis includes user preferences, the level of detail of the description and the relevance of the creation theme.

[0024] Optionally, the S3 specifically includes:

[0025] S31, extracting the creation style, detail requirements and content theme in the text description according to the target modality;

[0026] S32, constructing a generation input according to the extracted creation style, detail requirements and content theme, including the creation style, content details, theme information and parameters of the generation target;

[0027] S33, adjusting the format of the generation input according to the target modality type, if the target modality is an image or a video, converting the generation input into the parameter format required by the image or video generation model, if the target modality is an audio, converting the generation input into the parameter format required by the audio generation model;

[0028] S34, refining the user demand information in the generation input, the demand information including the style features of the image or video, the tone and rhythm requirements of the audio, and the emotional tendency of the creation theme;

[0029] S35, passing the adjusted generation input to the generation model as the input condition of the generation process.

[0030] Optionally, the S4 specifically includes:

[0031] S41, passing the generation input into the improved VQ-VAE-2 model, the improved VQ-VAE-2 model combining a long short-term memory network module and a recurrent autoencoder module;

[0032] S42, time series modeling of input data using a long short-term memory network module to handle temporal dependencies in image, video or audio content, generating feature representations for each time step;

[0033] S43, using a recurrent autoencoder module to encode the temporal dependencies in the time series data into a latent representation, and passing the latent representation to the improved VQ-VAE-2 model for generation;

[0034] S44, generating image, video or audio content based on the outputs of the long short-term memory network module and the recurrent autoencoder module, optimizing the temporal continuity and structural consistency of the time series data;

[0035] S45, processing the generated image, video or audio content to ensure that the details, transitions and quality of the content meet the requirements of the input description.

[0036] Optionally, the S42 specifically includes:

[0037] S421, receiving input data from step S41, the input data containing time series data in image, video or audio content;

[0038] S422, processing the input data step by step through the long short-term memory network module, the input data for each time step t being x t , x t represents the input data at time t;

[0039] S423, in the long short-term memory network module, the hidden state h t for each time step is calculated, the long short-term memory network processing input data through three main gate mechanisms, including: forget gate f t , input gate i t and output gate o t :

[0040]

[0041] Where f t is the forget gate, i t is the input gate, C t is the memory cell, h t is the hidden state at time t, W f ,W i ,W C ,W o are weight matrices, and b f ,b i ,b C ,b ois a bias term, σ is a sigmoid activation function, tanh is a hyperbolic tangent activation function, used to control the opening degree of each gate and the update of the memory cell;

[0042] S424, generating feature representations for each time step according to the output h of the long short-term memory network module t and passing these feature representations as latent representations to the improved VQ-VAE-2 model for further processing.

[0043] Optionally, the S43 specifically includes:

[0044] S431, receiving the output h of the long short-term memory network module passed from step S42 t , h t represents the hidden state at the t-th time;

[0045] S432, passing the hidden state h t as input to the recurrent autoencoder module, which includes an encoder and a decoder, the encoder being used to map the latent information of the time series data to a latent space, and the decoder being used to reconstruct the data from the latent space;

[0046] S433, in the encoder part, encoding the input h t through a recurrent neural network structure to generate a latent representation z t , and passing the latent representation z t to the decoder;

[0047] S434, in the decoder part, generating image, video or audio content based on the latent representation z t , the decoder optimizing the reconstruction result through a backpropagation algorithm to generate image, video or audio output;

[0048] S435, using the output of the recurrent autoencoder module to generate image, video or audio content, and optimizing the time continuity and structural consistency of the time series data;

[0049] S436, outputting the generated latent representation z t as an input condition for the generated image, video or audio content, and passing it to the improved VQ-VAE-2 model for further processing.

[0050] Optionally, the S5 specifically includes:

[0051] S51, receiving the latent representation z t of the generated image, video or audio content from step S4;

[0052] S52, according to the complexity of the generated content and the target modal type, applying an adaptive discrete space method to adjust the distribution of the latent space;

[0053] S53, applying a variational inference method to optimize the latent space, calculating the probability distribution of the latent representation and adjusting the distribution of the latent space;

[0054] S54, inputting the optimized latent representation z t to the next generation model as an input condition to generate the final image, video or audio content.

[0055] Optionally, the S52 specifically includes:

[0056] S521, according to the complexity of the generated content and the target modal type, selecting an adaptive discrete space method to adjust the latent space;

[0057] S522, adjusting the latent representation z t using the adaptive discrete space method; ′ t S523, according to the characteristics of the target modal, adjusting the coding structure of the quantization function Q, so that the latent representation z t t maintains structural consistency in the latent space; ′ t

[0058] S523, according to the characteristics of the target modal, adjusting the coding structure of the quantization function Q, so that the latent representation z ′ t maintains structural consistency in the latent space;

[0059] S524, inputting the adjusted latent representation z ′ t into the inference module used by the variational inference method.

[0060] Optionally, the S53 specifically includes:

[0061] S531, based on the input condition x and the latent representation z ′ t , constructing a variational inference framework, setting the true distribution p(z ′ t |x) and the variational distribution q(z ′ t |x) of the latent space;

[0062] S532, calculating the loss function in the variational inference, minimizing the Kullback-Leibler divergence between the variational distribution and the true distribution:

[0063]

[0064] ​​wherein KL is the Kullback-Leibler divergence, q(z ′ t is the variational distribution, p(z ′ t is the real distribution, x is the input condition, z ′ t is the discretized latent representation;

[0065] S533, according to the divergence calculation result, the parameter of the variational distribution is updated, and the probability structure of the latent space is optimized;

[0066] S534, output the optimized latent representation z ′ t and input into the generation model in the next step as the input condition.

[0067] The beneficial effects of the present application are:

[0068] The present application combines a large language model with an improved generation model to construct an automatic generation method for multi-modal data such as images, videos and audio, which not only supports static content generation, but also effectively processes time series data, adapting to the complex requirements of digital media works in terms of dynamics and structural integrity.

[0069] In terms of generation structure, the present application introduces a long short-term memory network and a recurrent autoencoder to model the temporal dependence in the generation process, thereby improving the consistency and coherence of the generated content in the time dimension. In terms of latent space control, the present application proposes an adaptive discrete space mechanism, combined with variational inference method to realize dynamic optimization of latent encoding, so that the model has stronger structural adjustment ability when facing different generation targets.

[0070] In addition, the present application takes DeepSeek large language model as the language understanding core, realizes the semantic consistency judgment between text input and generation result through semantic vector mapping and similarity calculation, and establishes a language-driven generation content control mechanism, which can judge and select the final generated image, video or audio content based on semantics.

[0071] In summary, the method proposed in the present application can cover multiple key links such as text analysis, latent modeling, time series generation and semantic alignment, forming a complete automatic generation process, and improving the comprehensive performance of digital media works in terms of structural expression, semantic matching and modal adaptation. BRIEF DESCRIPTION OF DRAWINGS

[0072] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and are meant to explain the present application but are not intended to limit the application. In the drawings:

[0073] Fig. 1 A flowchart of a digital media work automatic generation method based on a large language model according to the present application;

[0074] Fig. 2 An improved VQ-VAE-2 model structure diagram in a digital media work automatic generation method based on a large language model according to the present application;

[0075] Fig. 3 A structure diagram of a semantic consistency determination module in a digital media work automatic generation method based on a large language model according to the present application. DETAILED DESCRIPTION

[0076] The application will now be described in further detail with reference to the drawings. These drawings show only the essential features of the application and are therefore to be regarded only as a schematic illustration. The drawings are as follows:

[0077] REFERENCE Figs. 1-3 A digital media work automatic generation method based on a large language model, comprising the following steps:

[0078] S1, receiving a text description input by a user, using a DeepSeek large language model to analyze the text description and extract a creation style, content requirements and a target modality;

[0079] S2, determining the target modality according to the extracted creation information, and selecting a generation model corresponding to the target modality;

[0080] S3, combining the generation conditions with the key information in the text description according to the selected generation model to construct a corresponding generation input;

[0081] S4, inputting the generation input into an improved VQ-VAE-2 model, the improved VQ-VAE-2 model combining a long short-term memory network module and a recurrent autoencoder to process the time dependence in time series data, and generating image, video or audio content corresponding to the target modality;

[0082] S5, in the improved VQ-VAE-2 model, using an adaptive discrete space and a variational inference method to dynamically adjust the distribution of the latent space according to the data complexity, and optimizing the generated content;

[0083] S6, using a DeepSeek large language model to perform semantic consistency checking on the generated image, video or audio content;

[0084] S7, optimizing and adjusting the generated content according to the semantic consistency check result;

[0085] S8, outputting the generated digital media work, including image, video or audio file, back to the user.

[0086] The present application realizes a complete link from text description analysis to multi-modal content generation by constructing an automatic generation process based on a large language model, supporting unified modeling and generation control of images, videos and audio.

[0087] In the embodiment, S2 specifically includes:

[0088] S21, receiving the creation information extracted from S1, including creation style, content requirements and target mode;

[0089] S22, determining the target mode according to the creation information, including image, video and audio;

[0090] S23, if the target mode is image or video, selecting a generation model related to image or video generation for generating image or video content;

[0091] S24, if the target mode is audio, selecting a generation model related to audio generation for generating audio content;

[0092] S25, constructing a generation input according to the selected generation model and target mode, including creation style, content details and key information of generation target;

[0093] S26, adjusting the parameters of the generation input according to the creation style and content requirements, and the adjustment basis includes user preferences, description details and relevance of creation theme.

[0094] The present application realizes a complete link from text description analysis to multi-modal content generation by constructing an automatic generation process based on a large language model, supporting unified modeling and generation control of images, videos and audio.

[0095] In the embodiment, S3 specifically includes:

[0096] S31, extracting the creation style, detail requirements and content theme in the text description according to the target mode;

[0097] S32, constructing a generation input according to the extracted creation style, detail requirements and content theme, including creation style, content details, theme information and parameters of generation target;

[0098] S33, according to the target modal type, adjusting the format of the generated input, if the target modal is an image or a video, converting the generated input into the parameter format required by the image or video generation model, if the target modal is an audio, converting the generated input into the parameter format required by the audio generation model;

[0099] S34, in the generated input, refining the user demand information, the demand information including the style characteristics of the image or the video, the tone and rhythm requirements of the audio, and the emotional tendency of the creation theme;

[0100] S35, delivering the adjusted generated input to the generation model as the input condition of the generation process.

[0101] The present application can effectively convert the creation style, content details and theme parameters in the language information into the format required by the generation model by constructing a standardized process for generating input, ensuring the analyzability and executability of the language instruction in the generation process.

[0102] In the embodiment, the S4 specifically includes:

[0103] S41, delivering the generated input into the improved VQ-VAE-2 model, the improved VQ-VAE-2 model combining a long short-term memory network module and a recurrent autoencoder module;

[0104] S42, using the long short-term memory network module to model the time series of the input data, processing the time dependence in the image, video or audio content, and generating the feature representation of each time step;

[0105] S43, using the recurrent autoencoder module to perform recurrent encoding on the input data, encoding the time dependence in the time series data into a latent representation, and delivering the latent representation to the improved VQ-VAE-2 model for generation;

[0106] S44, generating the image, video or audio content according to the output of the long short-term memory network module and the recurrent autoencoder module, and optimizing the time continuity and structural consistency of the time series data;

[0107] S45, processing the generated image, video or audio content to ensure that the details, transitions and quality of the content meet the requirements of the input description.

[0108] The present application integrates long short-term memory network and recurrent autoencoder in the VQ-VAE-2 structure, so that the generation model can process the characteristics of time series data, and generate high-consistency multi-modal output content through structural modeling.

[0109] In the embodiment, the S42 specifically includes:

[0110] S421, receiving the input data from step S41, the input data containing time series data in image, video or audio content;

[0111] S422, processing the input data step by step through the long short-term memory network module, the input data of each time step t being x t , x t represents the input data at time t;

[0112] S423, in the long short-term memory network module, the hidden state h t of each time step is calculated, the long short-term memory network processes the input data through three main gating mechanisms, including: forget gate f t , input gate i t and output gate o t :

[0113]

[0114] Where f t is the forget gate, i t is the input gate, C t is the memory cell, h t is the hidden state at time t, W f , W i , W C , W o are weight matrices, b f , b i , b C , b o are bias terms, sigma is a sigmoid activation function, and tanh is a hyperbolic tangent activation function, used to control the opening degree of each gate and the update of the memory cell;

[0115] S424, according to the output h t of the long short-term memory network module, the feature representation of each time step is generated, and these feature representations are passed to the improved VQ-VAE-2 model as latent representations for further processing.

[0116] The present application uses long short-term memory network to model the dynamic state of input time series data, captures the relationship between historical information and current input through gating mechanism, generates time step feature representation, and provides foundation for latent representation construction.

[0117] In the embodiment, the S43 specifically comprises:

[0118] S431, receiving the output h t of the long short-term memory network module from step S42, h t represents the hidden state at time t;

[0119] S432, the hidden state h t As an input into the recurrent autoencoder module, the recurrent autoencoder module includes an encoder for mapping the latent information of the time series data to a latent space and a decoder for reconstructing the data from the latent space;

[0120] S433, in the encoder part, the input h t is encoded by a recurrent neural network structure to generate a latent representation z t , and the latent representation z t is passed to the decoder;

[0121] S434, in the decoder part, the latent representation z t is used to generate image, video or audio content, and the decoder optimizes the reconstruction result by a back propagation algorithm to generate image, video or audio output;

[0122] S435, the output of the recurrent autoencoder module is used to generate image, video or audio content, and the time continuity and structural consistency of the time series data are optimized;

[0123] S436, the generated latent representation z t is output as an input condition for the generated image, video or audio content, and is passed to the improved VQ-VAE-2 model for further processing.

[0124] The present application realizes the compression and reconstruction of the latent representation of time series data by the recurrent autoencoder, establishes the encoder-decoder structure, converts the hidden state into modal generated content, and enhances the structural modeling ability of the model for continuous data.

[0125] In the embodiment, the S5 specifically includes:

[0126] S51, receiving the latent representation z t of the generated image, video or audio content from step S4;

[0127] S52, according to the complexity of the generated content and the target modal type, an adaptive discrete space method is applied to adjust the distribution of the latent space;

[0128] S53, a variational inference method is applied to optimize the latent space, calculate the probability distribution of the latent representation and adjust the distribution of the latent space;

[0129] S54, the optimized latent representation z t is passed to the next generation model as an input condition to generate the final image, video or audio content.

[0130] The adaptive discrete space and variational inference mechanism are introduced to dynamically optimize the latent space in the generation process, and the adaptability of the latent representation to different modalities and complex tasks is enhanced.

[0131] In the embodiment, the S52 specifically includes:

[0132] S521, according to the complexity of the generated content and the target modality type, an adaptive discrete space method is selected to adjust the latent space;

[0133] S522, the latent representation z t is adjusted by applying a discrete quantization operation, and the discrete process is defined as z ′ t =Q(z t ), wherein z t is the original latent representation, z ′ t is the discrete latent representation, and Q is the quantization function;

[0134] S523, according to the characteristics of the target modality, the coding structure of the quantization function Q is adjusted to make the latent representation z ′ t maintain structural consistency in the latent space;

[0135] S524, the adjusted latent representation z ′ t is input into the inference module used by the variational inference method.

[0136] By applying the adaptive discrete space method, the adjustable discrete quantization mechanism is introduced in the latent space, so that the latent representation structure can be adjusted according to the feature requirements of different generation tasks.

[0137] In the embodiment, the S53 specifically includes:

[0138] S531, based on the input condition x and the latent representation z ′ t , a variational inference framework is constructed, and the real distribution p(z ′ t |x) and the variational distribution q(z ′ t |x) of the latent space are set;

[0139] S532, the loss function in the variational inference is calculated, and the Kullback-Leibler divergence between the variational distribution and the real distribution is minimized:

[0140]

[0141] wherein KL is the Kullback-Leibler divergence, q(z ′ t is the variational distribution, p(z ′ t is the real distribution, x is the input condition, z ′ t is the discretized latent representation;

[0142] S533, according to the divergence calculation result, performing parameter updating on the variational distribution to optimize the probability structure of the latent space;

[0143] S534, outputting the optimized latent representation z ′ t and inputting it as the input condition into the generation model in the next step.

[0144] The present application optimizes the distance relationship between the variational distribution and the real distribution by constructing a variational inference framework, improves the probability expression ability of the latent space, and provides stable latent support structure for the subsequent generation process.

[0145] Embodiment 1

[0146] In order to verify the feasibility of the present application in implementation, the present application is applied to a short video and text generation platform built by a digital content creation company in Chengdu. The platform faces content creators in the fields of film and television production, advertising design, and self-media operation, and provides tool services for automatically generating images, videos, and audios based on text input. In the past, the platform has relied heavily on manual drawing and editing in content production, resulting in slow generation speed, high personnel cost, and unresponsive creative ideas.

[0147] In this deployment, the platform integrates an automatic generation method of digital media works based on a large language model proposed by the present application. After receiving the natural language description input by the user, the system first analyzes the text content through the DeepSeek large language model, extracts the creative style, theme semantics, content intention, and target modal type (image, video, or audio). Then, the system automatically selects the appropriate generation model structure according to the modal type, and inputs the structured information into the improved VQ-VAE-2 model. The model combines long short-term memory networks to process time sequence features, and uses a recurrent autoencoder to encode and decode the latent structure.

[0148] To further enhance the expression ability and adaptability of the model, the generation process introduces a self-adaptive discrete space mechanism to dynamically adjust the quantization structure of the latent representation according to the complexity of the content. At the same time, the system optimizes the distribution structure of the latent space based on the variational inference method, so as to more accurately fit the semantic content. After generation is completed, the platform calls the DeepSeek model again to extract the semantic vectors of the original text and the generated content, respectively, and checks the semantic consistency by calculating the cosine similarity, and eliminates the mismatched results or automatically feeds back for optimization and regeneration.

[0149] In actual testing, we selected 10 representative text inputs, covering natural scenery, fantasy scenes, life stills, and urban themes. Each input was generated by the platform through the whole process, and the generation mode, time consumption, semantic consistency score, and manual evaluation satisfaction were recorded. Table 1 is a summary of the test results:

[0150] Table 1 Digital media work automatic generation test results

[0151]

[0152]

[0153] As can be seen from Table 1, in the image task, the average generation time is 2.33 seconds, the semantic consistency score is concentrated between 0.90 and 0.94, and the manual evaluation satisfaction is 4.72 on average; in the video task, the generation time is 4.8 seconds and 5.2 seconds respectively, the semantic consistency score is also stable at 0.88 and 0.89, and the manual score is 4.5 and 4.4 respectively. The results show that the present invention not only significantly outperforms traditional methods in generation speed, but also has good performance in semantic matching and structural coherence, and can meet the needs of users for high-quality multi-modal generated content.

[0154] In summary, the present invention successfully solves the difficult problems of existing generation techniques in multi-modal matching, time series modeling, and semantic control in real production environment, and has significant practical value and industrial promotion potential.

[0155] The above describes only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto, and any person skilled in the art can make equivalent substitutions or changes to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered within the protection scope of the present invention.

Claims

1. A method for automatically generating digital media works based on large language models, characterized in that, Comprising the following steps: S1, receiving user input text description, using DeepSeek large language model to analyze text description, extracting creative style, content requirements and target modality; S2, according to the extracted creative information, determine the target modality, and select the generation model corresponding to the target modality; S3, according to the selected generation model, combine the generation condition with the key information in the text description to construct the corresponding generation input; S4, the improved VQ-VAE-2 model is used to input the generation input, the improved VQ-VAE-2 model combines long short-term memory network module and recurrent autoencoder to process the time dependence in time series data, and generates image, video or audio content corresponding to the target modality; S5, in the improved VQ-VAE-2 model, the adaptive discrete space and variational inference method are used to dynamically adjust the distribution of latent space according to the data complexity, and the generated content is optimized; S6, using DeepSeek large language model to check the semantic consistency of the generated image, video or audio content; S7, according to the semantic consistency checking result, the generated content is optimized and adjusted; S8, output the generated digital media works, including image, video or audio file, return to the user.

2. The method of claim 1, wherein the method further comprises: The S2 specifically comprises: S21, receiving the creative information extracted from S1 step, including creative style, content requirements and target modality; S22, according to the creative information, determine the target modality, including image, video and audio; S23, if the target modality is image or video, select the generation model related to image or video generation for generating image or video content; S24, if the target modality is audio, select the generation model related to audio generation for generating audio content; S25, according to the selected generation model and target modality, construct the generation input, including creative style, content details and key information of generation target; S26, according to the creative style and content requirements, adjust the parameters of the generation input, and the adjustment basis includes user preference, description detail and relevance of creative theme.

3. The method of claim 1, wherein the method further comprises: The S3 specifically comprises: S31, according to the target modality, extract the creative style, detail requirement and content theme in the text description; S32, according to the extracted creative style, detail requirement and content theme, construct the generation input, including creative style, content detail, theme information and parameter of generation target; S33, according to the target modality type, adjust the format of the generation input, if the target modality is image or video, convert the generation input into the parameter format required by the image or video generation model, if the target modality is audio, convert the generation input into the parameter format required by the audio generation model; S34, in the generation input, refine the user demand information, the demand information includes the style characteristics of image or video, the tone and rhythm requirements of audio, the emotional tendency of creative theme; S35, the adjusted generation input is transmitted to the generation model as the input condition of the generation process.

4. The method of claim 1, wherein the method further comprises: The S4 specifically comprises: S41, input the generated input into the improved VQ-VAE-2 model, which combines a long short-term memory network module and a recurrent autoencoder module; S42, use the long short-term memory network module to model the time series of the input data, process the time dependence in the image, video or audio content, and generate a feature representation for each time step; S43, use the recurrent autoencoder module to encode the time dependence in the time series data into a latent representation, and pass the latent representation to the improved VQ-VAE-2 model for generation; S44, generate the image, video or audio content according to the output of the long short-term memory network module and the recurrent autoencoder module, and optimize the time continuity and structural consistency of the time series data; S45, process the generated image, video or audio content to ensure that the details, transitions and quality of the content meet the requirements of the input description.

5. The method of claim 4, wherein, The S42 specifically includes: S421, receiving the input data from step S41, the input data containing time series data in the image, video or audio content; S422, step-by-step processing of the input data is performed by the long short-term memory network module, the input data of each time step t is x t , x t represents the input data at the t th time S423、In the long short-term memory network module, the hidden state h of each time step is calculated t The long short-term memory network processes the input data through three main gating mechanisms, including: the forget gate f t , the input gate i t and the output gate o t : wherein f t is a forget gate, i t is an input gate, C t is a memory cell, h t is a hidden state at the t-th time, W f ,W i ,W C ,W o is a weight matrix, b f ,b i ,b C ,b o is a bias term, and σ is a Sigmoid activation function and tanh is a hyperbolic tangent activation function. S424, generating feature representations for each time step from the output h of the long short-term memory network module t and passing these feature representations as latent representations to the improved VQ-VAE-2 model for further processing.

6. The method of claim 4, wherein the method further comprises: The S43 specifically includes: S431, Receive the output h from the Long Short-Term Memory network module passed in step S42. t h t Let represent the hidden state at time t; S432, the hidden state h t As an input into the recurrent autoencoder module, the recurrent autoencoder module includes an encoder for mapping the latent information of the time series data to a latent space and a decoder for reconstructing the data from the latent space. S433、At the encoder part, the input h t is encoded by a recurrent neural network structure to generate a latent representation z t , and the latent representation z t is passed to the decoder. S434、At the decoder part, based on the latent representation z t generating image, video or audio content, the decoder optimizes the reconstruction result by a backpropagation algorithm, generating an image, video or audio output; S435, generating the image, video or audio content using the output of the recurrent autoencoder module, and optimizing the time continuity and structural consistency of the time series data; S436、output the generated latent representation z t As input conditions for the generated image, video, or audio content, passed to the improved VQ-VAE-2 model for further processing.

7. The method of claim 1, wherein the method further comprises: The S5 specifically includes: S51, receiving the latent representation z of the image, video or audio content generated from step S4 t ; S52, according to the complexity of the generated content and the target modal type, apply an adaptive discrete space method to adjust the distribution of the latent space; S53, apply a variational inference method to optimize the latent space, calculate the probability distribution of the latent representation and adjust the distribution of the latent space; S54, the optimized latent representation z t As input conditions to the next step generation model, the final image, video or audio content is generated.

8. The method of claim 7, wherein the method further comprises: The S52 specifically includes: S521, according to the complexity of the generated content and the target modal type, select an adaptive discrete space method to adjust the latent space; S522, on a potential representation z t Applying a discrete quantization operation, the discretization process is defined as z ′ t = Q(z t ), where z t is the original latent representation, z ′ t is the discretized latent representation, and Q is the quantization function; S523、According to the characteristics of the target modality, adjust the encoding structure of the quantization function Q, so that the latent representation z ′ t Maintain structural consistency in the latent space; S524, adjusting the latent representation z ′ t into an inference module used by the variational inference method.

9. The method of claim 7, wherein the method further comprises: The S53 specifically includes: S531, based on the input condition x and the latent representation z ′ t , a variational inference framework is constructed, a true distribution p(z ′ t |x) of the latent space and a variational distribution q(z ′ t |x) are set; S532, calculate the loss function in variational inference, minimize the Kullback-Leibler divergence between the variational distribution and the true distribution: where KL is the Kullback-Leibler divergence, q(z ′ t ′ t ′ t is the discrete latent representation;​​ S533, according to the divergence calculation result, update the parameters of the variational distribution, and optimize the probability structure of the latent space; S534, output the optimized latent representation z ′ t and as input conditions into the generative model in the next step.

Images