Image and video reconstruction method and system based on diffusion model, terminal and storage medium

Abstract

The invention discloses an image and video reconstruction method and system based on a diffusion model, a terminal and a storage medium, and the method comprises the steps: obtaining original image data and original text description, inputting the original image data and original text description into the diffusion model, and obtaining noise data and intermediate features; determining a time step prediction model, and performing reverse generation processing on the noise data, the intermediate features and the original text description through the time step prediction model and a diffusion model to obtain initial image data; updating the time step prediction model according to the initial image data to obtain an updated time step prediction model; and obtaining an updated text description, and performing reverse generation processing on the noise data, the intermediate features and the updated text description through the updated time step prediction model and the diffusion model to obtain target image data. According to the method, the reconstruction capability of the diffusion model can be effectively improved, and accurate reconstruction of original image data is realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an image and video reconstruction method, system, terminal and computer-readable storage medium based on a diffusion model. Background Art

[0002] With the rapid development of digital content creation, the demand for image and video editing is growing. Traditional image and video editing methods usually rely on professional software. Although these software are powerful, they require users to have certain professional skills and experience, and the learning cost is high. In addition to the time-consuming and cumbersome manual operation required to handle complex editing tasks, these tools also have certain limitations in achieving certain complex creative effects (such as style transfer, content replacement, etc.). In recent years, with the rapid development of image generation models and video generation models, image and video editing methods based on diffusion models have gradually become a research hotspot.

[0003] However, the existing technology faces the problem of content consistency when using diffusion models for image and video editing. In the process of editing an image or video, especially when modifying a local area of ​​an image or video, the generated content may not completely match the style, structure or context of the original image or video, thus failing to meet the user's requirements for image and video editing.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide an image and video reconstruction method, system, terminal and computer-readable storage medium based on a diffusion model, aiming to solve the problem that when the existing technology uses a diffusion model to edit images or videos, the generated content does not completely match the style, structure or context of the original image or video, thereby failing to meet the user's requirements for image and video editing.

[0006] To achieve the above object, the present invention provides an image and video reconstruction method based on a diffusion model, and the image and video reconstruction method based on a diffusion model comprises the following steps: Acquire original image data and original text description, and input the original image data and the original text description into a diffusion model to obtain noise data and intermediate features; Determine a time step prediction model, and perform reverse generation processing on the noise data, the intermediate features, and the original text description through the time step prediction model and the diffusion model to obtain initial image data; Calculating a reconstruction error between the original image data and the initial image data, and updating the time step prediction model according to the reconstruction error to obtain an updated time step prediction model; An updated text description is obtained, and the noise data, the intermediate features and the updated text description are subjected to reverse generation processing through the updated time step prediction model and the diffusion model to obtain target image data.

[0007] Optionally, the image and video reconstruction method based on the diffusion model, wherein the obtaining of original image data and original text description, and inputting the original image data and the original text description into the diffusion model to obtain noise data and intermediate features, specifically includes: Acquire original image data and original text description, and input the original image data and the original text description into a diffusion model; Performing forward diffusion processing on the original image data and the original text description through the diffusion model to obtain noise data; A preset time step is determined, and a key-value cache of an attention mechanism in the diffusion model is extracted according to the preset time step, and an intermediate feature corresponding to the preset time step is obtained according to the key-value cache.

[0008] Optionally, the image and video reconstruction method based on the diffusion model, wherein the determining of the time step prediction model and performing reverse generation processing on the noise data, the intermediate features and the original text description through the time step prediction model and the diffusion model to obtain the initial image data, specifically includes: Inputting the noise data, the intermediate features and the original text description into the diffusion model, and performing reverse generation processing on the noise data, the intermediate features and the original text description through the diffusion model; Determine a time step prediction model, and perform prediction processing on each time step in the reverse generation process through the time step prediction model; When any time step in the reverse generation process is 0, the reverse generation process is completed, and the diffusion model outputs initial image data.

[0009] Optionally, the image and video reconstruction method based on the diffusion model, wherein the step of calculating the reconstruction error between the original image data and the initial image data, and updating the time step prediction model according to the reconstruction error to obtain the updated time step prediction model, specifically comprises: Calculating a reconstruction error between the original image data and the initial image data, and determining whether the reconstruction error is greater than or equal to a preset threshold; If so, the time step prediction model is updated to obtain an updated time step prediction model.

[0010] Optionally, the image and video reconstruction method based on the diffusion model, wherein the updating of the time step prediction model to obtain the updated time step prediction model specifically includes: Obtaining a negative number of the reconstruction error, and using the negative number as a reward signal; Historical generation data is obtained, and the time step prediction model is updated according to the historical generation data and the reward signal to obtain an updated time step prediction model.

[0011] Optionally, the image and video reconstruction method based on the diffusion model, wherein the reconstruction error between the original image data and the initial image data is calculated, and the time step prediction model is updated according to the reconstruction error to obtain an updated time step prediction model, further comprises: The noise data, the intermediate features and the original text description are iteratively reversely generated by using the update time step prediction model and the diffusion model until the reconstruction error is less than the preset threshold.

[0012] Optionally, the image and video reconstruction method based on the diffusion model, wherein the acquiring of the updated text description and performing reverse generation processing on the noise data, the intermediate features and the updated text description through the updated time step prediction model and the diffusion model to obtain the target image data, specifically includes: Determine user needs, and perform text update processing on the original text description according to the user needs to obtain an updated text description, wherein the text update processing includes keyword replacement and content detail modification; The updated text description, the noise data and the intermediate features are input into the diffusion model again, and the updated text description, the noise data and the intermediate features are reversely generated through the updated time step prediction model and the diffusion model to obtain target image data.

[0013] In addition, to achieve the above-mentioned purpose, the present invention further provides an image and video reconstruction system based on a diffusion model, wherein the image and video reconstruction system based on a diffusion model comprises: A forward diffusion processing module, used for acquiring original image data and original text description, and inputting the original image data and the original text description into a diffusion model to obtain noise data and intermediate features; A time step prediction model processing module is used to determine a time step prediction model, and perform reverse generation processing on the noise data, the intermediate features and the original text description through the time step prediction model and the diffusion model to obtain initial image data; A time step prediction model updating module, used for calculating a reconstruction error between the original image data and the initial image data, and updating the time step prediction model according to the reconstruction error to obtain an updated time step prediction model; The reverse generation processing module is used to obtain the updated text description, and perform reverse generation processing on the noise data, the intermediate features and the updated text description through the updated time step prediction model and the diffusion model to obtain the target image data.

[0014] In the present invention, the original image data and the original text description are obtained, and the original image data and the original text description are input into the diffusion model to obtain noise data and intermediate features; the time step prediction model is determined, and the noise data, the intermediate features and the original text description are reversely generated through the time step prediction model and the diffusion model to obtain the initial image data; the reconstruction error between the original image data and the initial image data is calculated, and the time step prediction model is updated according to the reconstruction error to obtain the updated time step prediction model; the updated text description is obtained, and the noise data, the intermediate features and the updated text description are reversely generated through the updated time step prediction model and the diffusion model to obtain the target image data. The present invention uses the time step prediction model to assist the diffusion model in reverse generation, which can effectively improve the reconstruction ability of the diffusion model and reduce the reconstruction error of the diffusion model in the reverse generation process, thereby ensuring that the generated target image data is more consistent with the original image data, and effectively improving the reconstruction effect of the original image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of a preferred embodiment of the image and video reconstruction method based on the diffusion model of the present invention; Figure 2 It is a schematic diagram of the overall structure implementation process of a preferred embodiment of the image and video reconstruction method based on the diffusion model of the present invention; Figure 3 It is a structural diagram of a preferred embodiment of the image and video reconstruction system based on the diffusion model of the present invention; Figure 4 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0017] With the rapid development of digital content creation, the demand for image and video editing is growing. Traditional editing methods usually rely on professional software such as Adobe Photoshop and Premiere (Adobe Photoshop and Premiere are both image processing software). Although these software are powerful, they require users to have certain professional skills and experience, and the learning cost is high. In addition to requiring a lot of manual operations when handling complex editing tasks, which is time-consuming and tedious, these tools also have certain limitations when achieving certain complex creative effects (such as style transfer, content replacement, etc.).

[0018] In recent years, with the rapid development of image generation models and video generation models, image and video editing methods based on diffusion models have gradually become a research hotspot. Diffusion models are a type of generative models that work by gradually adding noise to data and learning how to reversely denoise to generate new data. This process mainly includes two stages: the forward diffusion stage and the reverse generation stage. In the forward diffusion stage, the diffusion model starts with the original data, gradually adds noise, and generates a series of gradually degraded data; in the reverse generation stage, the diffusion model starts from the noisy data, gradually denoises and restores the original data.

[0019] However, when using diffusion models for image and video editing, there is a problem of content consistency. During the editing process, especially when modifying local areas of an image or video, the generated content may not fully match the style, structure or context of the original image, and thus cannot meet the requirements of image and video editing. This is because each part of the content generated by the diffusion model is gradually denoised based on noise data and control conditions, and it is difficult to ensure consistency with the details of the original image or video, especially in complex backgrounds or diverse scenes. However, due to the multi-step denoising operation and the introduction of text descriptions, the generated image is often difficult to be consistent with the original image, that is, the diffusion model has the weakness of poor reconstruction ability, which limits its editing ability on images and videos.

[0020] Image and video editing methods based on diffusion models have the following two main shortcomings: 1. Lack of adaptive capabilities: Most existing methods rely on a unified, deterministic training-free strategy and cannot be adaptively adjusted according to different actual editing scenarios. This means that they may still have unavoidable reconstruction errors when processing different types of images or videos. Although these methods provide a certain degree of improvement, they fail to perform personalized optimization for specific editing tasks, which limits their effectiveness in practical applications. 2. Failure to utilize the training capabilities of neural networks: Existing technologies do not fully utilize the potential of neural networks to adaptively solve the editing needs of different scenarios through parameter training. Due to the lack of a dynamic adjustment mechanism based on training, these methods cannot effectively reduce or eliminate reconstruction errors, thereby affecting the performance and generation effect of the diffusion model in complex editing tasks. They fail to fully utilize the powerful capabilities of neural networks to improve generation quality through learning optimization and reducing reconstruction errors.

[0021] To solve the above problems, the present invention provides an image and video editing method that improves the reconstruction capability of the diffusion model. Through a training-based method, the diffusion model can adaptively adjust the parameters of the neural network according to the actual image and video editing scenarios, thereby more accurately reconstructing the original image or video content, and significantly improving the editing effect of the diffusion model. The present invention optimizes the reconstruction capability of the diffusion model through a training process, so that it can cope with editing needs in different scenarios, solves the reconstruction error problem existing in existing methods in diversified editing tasks, and provides a more efficient and flexible solution for the field of image and video editing.

[0022] The image and video reconstruction method based on the diffusion model described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the image and video reconstruction method based on the diffusion model includes the following steps: Step S10: Obtain original image data and original text description, and input the original image data and the original text description into a diffusion model to obtain noise data and intermediate features. The original image data may refer to image data or video data.

[0023] The technical fields involved in the present invention include: 1. AIGC technology (Artificial Intelligence Generated Content): using artificial intelligence algorithms, especially deep learning and natural language processing technology, to automatically generate various forms of content, including text, images, audio and video. AIGC uses a large amount of data to train the model, so that the computer can simulate the process of human creation, thereby automatically generating creative and personalized content. 2. Diffusion Model Optimization Technology: Through a series of methods and strategies, the effect and efficiency of the diffusion model in the generation task are improved. The diffusion model generates high-quality samples from the noise by simulating the process of gradual noise addition and denoising. It is widely used in image generation, image restoration, super-resolution and text generation. The core goal of the optimization technology is to improve the generation quality, accelerate the generation process, reduce the computing cost, and enhance the stability of the model. Common optimization techniques include network architecture improvement, noise scheduling strategy optimization, learning rate adjustment, pre-training and fine-tuning, sampling strategy optimization and regularization methods. These technologies help the diffusion model achieve higher results in multiple generation tasks and make it more efficient and practical in practical applications. 3. Intelligent Image and Video Editing Systems: Utilize artificial intelligence and deep learning technologies to automate or enhance traditional image and video editing processes. Through intelligent algorithms, these systems can automatically repair image defects, enhance image quality, perform style conversion, and perform object recognition and editing. They can also intelligently edit videos, automatically generate background music, and add special effects in real time, significantly improving creative efficiency and reducing editing difficulty. These technologies are widely used in social media, advertising, entertainment, and art creation, promoting the automation and personalization of content production. 4. Real-time Editing & Acceleration: Ensure real-time processing and editing of multimedia content such as images and videos through hardware and software optimization. These technologies rely on GPU acceleration, deep learning inference optimization, cloud computing, and edge computing to improve editing efficiency and reduce latency. For example, through GPU and deep learning acceleration, the processing of images and videos can be completed quickly, supporting real-time special effects applications, dynamic adjustments, and efficient rendering. Real-time editing and computing acceleration technologies are widely used in video production, live broadcasting, virtual reality, and other fields, greatly improving the creation and interactive experience.

[0024] like Figure 2As shown, the image and video editing using the diffusion model in the present invention is mainly divided into two steps: first, the real image or video is converted into noise data through a forward diffusion process, and the intermediate features of the diffusion model at each time step are retained in this process; when the time step prediction model is updated, the text description is adjusted according to the user's editing preference, the edited text description is input into the diffusion model together with the converted noise data, and the previously retained intermediate features are injected at each time step of the reverse generation process, thereby generating an edited image or video.

[0025] It is understandable that the diffusion model can achieve good image and video editing effects only if it has strong reconstruction capabilities. For example, in the reverse generation process of the diffusion model, the input text is the unedited original text description, and the generated image or video should be consistent with the original content. Only on the basis of perfect reconstruction of content consistent with the original image or video can the text description be flexibly edited and the feature injection method be designed.

[0026] Specifically, original image data and original text description are obtained, and the original image data and the original text description are input into a diffusion model; forward diffusion processing is performed on the original image data and the original text description through the diffusion model to obtain noise data; a preset time step is determined, and a key-value cache of an attention mechanism in the diffusion model is extracted according to the preset time step, and an intermediate feature corresponding to the preset time step is obtained according to the key-value cache.

[0027] The method to improve the reconstruction capability of the diffusion model can be optimized from two perspectives. From the perspective of the forward diffusion process, the noise added in the forward diffusion stage is predicted by the current noise and text description input into the model. This ensures the consistency of the noise added in the forward diffusion stage and the noise removed in the reverse generation stage, because the noise is predicted by the model. At the same time, the intermediate features of the model in the forward process can be injected into the model in the reverse generation process, and the consistency of the generated image and the original image is promoted through the consistency of the model features. From the perspective of the reverse generation process, the inverse solution process of the diffusion model is optimized through mathematical derivation, and the error of multi-step denoising is gradually reduced.

[0028] First, the original image or original video (i.e., the original image data in the present invention) is input into the diffusion model and noise data and intermediate features are generated through forward diffusion. The specific process is as follows: Input the original image or video and original text description , noise data is generated iteratively through the forward diffusion process At a specific time step (Among them, is the time step, , is the number of time steps), extracting intermediate features from the diffusion model (i.e., the KV cache of the attention mechanism in the diffusion model), which reflects the latent variable structural information.

[0029] It is understandable that the process of forward expansion as follows: (Replace the original image or video Assign to time step Hidden variables ,in, For the time steps, is the time step corresponding hidden variables); : ;in, express The order of time step values ​​is To update the hidden variables corresponding to the time step, is the hidden variable corresponding to the time step, is the time step, To update the time step, is the diffusion model, is the original text description (update time step The corresponding hidden variable is ); ; (Noise generated by the forward diffusion process is the time step Hidden variables ).

[0030] Step S20, determining a time step prediction model, and performing reverse generation processing on the noise data, the intermediate features, and the original text description through the time step prediction model and the diffusion model to obtain initial image data.

[0031] Unlike the method of fine-tuning the diffusion model parameters, the present invention focuses on designing an additional lightweight model to predict the time steps in the reverse iterative generation process. The core idea of ​​this design is that fine-tuning the diffusion model may cause the original model to lose its ability to generate images or videos, especially after fine-tuning, the model may overfit to a specific editing task and lose its versatility in general generation tasks. In addition, models with a large number of fine-tuning parameters usually require a large amount of training data sets, but in actual image and video editing tasks, usually only a single image or a single video is edited, and the amount of data is far from enough to support large-scale fine-tuning.

[0032] Therefore, the present invention proposes to design a lightweight additional model (i.e., a time step prediction model) to specifically predict the time steps in the iteration process. The time step prediction model is optimized only for specific editing tasks, and does not require retraining the entire diffusion model, nor does it affect the general generation capability of the model. In this way, the evolution of each step in the editing process can be accurately controlled without interfering with the generation capability of the original model, thereby improving the reconstruction capability of the diffusion model.

[0033] Specifically, the noise data, the intermediate features and the original text description are input into the diffusion model, and the noise data, the intermediate features and the original text description are reversely generated through the diffusion model; a time step prediction model is determined, and each time step in the reverse generation process is predicted through the time step prediction model (i.e., the previously retained intermediate features are injected into each time step of the reverse generation process to generate an edited image or video); when any time step in the reverse generation process is 0, the reverse generation process is completed, and the diffusion model outputs the initial image data.

[0034] However, due to the approximation of the update iteration, that is, the update iteration Should be converted to , the diffusion model cannot reconstruct the original image or video well due to error accumulation.

[0035] Among them, the reverse generation process ( )as follows: (The noise data Assign to time step Hidden variables ); : ; (Update time step The corresponding hidden variable ); (The image or video generated by the reverse generation process is a time step Hidden variables ).

[0036] In order to reduce the reconstruction error, the present invention Perform Taylor expansion and convert the iterative formula By changing the formula to take the second-order term, a better reconstruction effect is obtained. However, the iterative formula is still an approximate formula, and there is still cumulative error in the reverse generation process of the diffusion model. Therefore, although the training-free method of image and video editing using the diffusion model can reduce the reconstruction error and improve the generation effect to a certain extent, it still relies on the approximate formula in essence and cannot completely eliminate the cumulative error, which limits the accuracy of the final reconstruction effect. In addition, because these methods do not take into account the diversity and complexity of actual image and video editing scenarios in their design, the trained models usually cannot generalize well to new and unseen editing tasks.

[0037] In order to overcome these problems, the present invention introduces a training mechanism, which can enable the diffusion model to automatically adjust parameters according to different editing scenarios, thereby reducing reconstruction errors and improving the generalization ability of the model in various practical applications.

[0038] The present invention performs reverse generation based on noise data and text description, and the specific implementation process is as follows: , original text description and intermediate features Input the diffusion model and perform the reverse generation process. The reverse generation is performed by iterating the mathematical expression Progressive denoising to restore images or videos , where the KV cache of the attention mechanism in the diffusion model consists of intermediate features Instead. In the reverse generation of each time step, the next time step Prediction model by time step Dynamic prediction is obtained. is the prediction model parameter. When it decreases to 0, an image or video is generated In particular, the noisy data and intermediate features The input to the diffusion model is for consistency. It is the completely noisy data generated by forward diffusion, which serves as the starting point for reverse generation, ensuring that the generation process starts from a noise distribution related to the original data, thereby maintaining content consistency; The structural information of the diffusion process is captured as an additional input to constrain the generation process, so that In details with to align.

[0039] Step S30, calculating a reconstruction error between the original image data and the initial image data, and updating the time step prediction model according to the reconstruction error to obtain an updated time step prediction model.

[0040] Specifically, the reconstruction error between the original image data and the initial image data is calculated, and it is determined whether the reconstruction error is greater than or equal to a preset threshold; if so, the inverse of the reconstruction error is obtained, and the inverse is used as a reward signal; the historical generation data is obtained, and the time step prediction model is updated according to the historical generation data and the reward signal to obtain an updated time step prediction model. Furthermore, the noise data, the intermediate features, and the original text description are iteratively reversely generated by the updated time step prediction model and the diffusion model until the reconstruction error is less than the preset threshold.

[0041] Furthermore, the present invention introduces the RLOO reinforcement learning training method, which regards the opposite of the reconstruction error as a reward signal and the entire time step sequence as an action sequence, thereby optimizing the parameters of the time step prediction model. Through the reinforcement learning training process, the model can adaptively adjust the time step prediction, thereby minimizing the reconstruction error in each iterative step and improving the consistency of the final generated content.

[0042] like Figure 2 As shown, the present invention evaluates the reconstruction error and determines the subsequent steps. The specific implementation process is as follows: 1. Calculate the reconstruction error between the generated image or video and the original image or video , is a square of the two-norm, indicating the difference between the two. 2. If the reconstruction error Below the preset threshold (generally set to 0.01), then jump directly to step S40; if the reconstruction error Above the preset threshold , the time step prediction model is optimized through reinforcement learning.

[0043] The specific implementation steps of optimizing the time step prediction model through reinforcement learning are as follows: 1. Use the opposite of the reconstruction error as the reward signal , defined as the reward signal 2. Use the RLOO method to optimize the time step prediction model parameters using historical generation data and reward signals The optimization goal is ,in, To optimize the goal, 3. Return to the step of inputting the noise data, the intermediate features and the original text description into the diffusion model, and performing reverse generation processing according to the diffusion model and the time step prediction model (the updated time step prediction model is used at this time), and perform a new round of reverse generation iteration.

[0044] Step S40, obtaining an updated text description, and performing reverse generation processing on the noise data, the intermediate features and the updated text description through the updated time step prediction model and the diffusion model to obtain target image data.

[0045] In the reverse generation stage, users can provide additional control conditions (such as text descriptions) for the diffusion model to make the generation process more flexible and controllable. This enables the diffusion model to generate diverse, high-quality image or video content, and users only need to guide the generation effect through simple text descriptions. This flexibility and efficiency make the diffusion model show great potential in the field of digital content creation and editing, especially when achieving complex creative effects, it is more convenient and efficient than traditional editing tools.

[0046] Specifically, user needs are determined, and text update processing is performed on the original text description according to the user needs to obtain an updated text description, wherein the text update processing includes keyword replacement and content detail modification; the updated text description, the noise data and the intermediate features are input into the diffusion model again, and reverse generation processing is performed on the updated text description, the noise data and the intermediate features through the update time step prediction model and the diffusion model to obtain target image data.

[0047] The present invention can describe the original text according to user needs. Make adjustments to generate the edited text description (i.e., updating the text description in the present invention), editing includes keyword replacement and / or content detail modification.

[0048] like Figure 2 As shown, the present invention generates new results based on the edited text description, and the specific implementation process is as follows: , edited text description and intermediate features Input the diffusion model again and perform reverse generation to obtain the edited image or video. At this time, the next time step The optimized time step prediction model The prediction is made and the edited image or video is finally output .

[0049] Beneficial effects of the present invention: 1. The present invention designs a lightweight time-step prediction model and combines it with reinforcement learning optimization. The model significantly improves the reconstruction capability of the diffusion model and achieves higher-precision reconstruction of the original content by reducing the accumulated error in the reverse generation process. Compared with traditional methods, it can more accurately control each generation step, avoid error accumulation in the multi-step denoising process, and ensure that the generated image or video content is more consistent with the original data.

[0050] 2. Adjustment of the adaptive generation process: The present invention has adaptive capabilities and can adjust the generation process according to different editing tasks, so as to better adapt to diverse editing scenarios. Through this adaptive mechanism, the present invention effectively avoids the reconstruction errors that occur in complex scenarios in the prior art, can handle different types of image or video editing tasks, and ensure the consistency and high quality of the generated content.

[0051] 3. No need to change the original diffusion model: The present invention does not require any changes to the original diffusion model, and retains the general performance of the diffusion model in tasks such as text-to-image or text-to-video. This feature enables the present invention to improve the performance of the model in actual editing tasks without changing the original model architecture, while taking into account flexibility and effectiveness.

[0052] 4. Low training data requirements and strong applicability: Compared with traditional methods, the present invention has lower requirements for training data, can be effectively trained under limited data conditions, and can quickly adapt to specific scenarios or tasks. It can flexibly adapt to personalized editing tasks of a single image or a single video, and can still maintain good performance in a small sample environment, and has a wide range of applications.

[0053] In summary, the innovative features of the present invention include: 1. The present invention designs a lightweight time-step prediction model for optimizing the reverse generation process of the diffusion model. By predicting the time-step sequence in the iterative process, the cumulative error in multi-step denoising is reduced, thereby improving the accuracy and consistency of the generated content; the present invention adopts reinforcement learning to optimize the time-step prediction model, takes the time-step sequence as the action sequence, and uses the opposite number of the reconstruction error as the reward signal for training, thereby realizing the function of dynamically adjusting the time-step sequence according to different editing tasks; the present invention minimizes the reconstruction error in each generation step through the above-mentioned dynamic adjustment mechanism, thereby significantly improving the quality of the final generation result.

[0054] 2. The present invention sets up an innovative process for adaptively adjusting the generation process: the present invention dynamically adjusts the generation process according to the actual editing scenario, integrates time step prediction, intermediate feature injection and text description guidance, to ensure that the generation results are highly consistent with user needs in complex editing tasks (such as style transfer, local modification); the present invention automatically improves the generation quality through an iterative optimization mechanism combined with threshold judgment and parameter update; the present invention can enhance the model performance in a variety of scenarios, so that the generated images or videos are highly consistent with the real data.

[0055] 3. Uniqueness of the technical combination: The present invention combines the diffusion model with a lightweight time-step prediction model, and solves the reconstruction error problem in traditional methods through a training mechanism; the present invention does not require large-scale fine-tuning of the original diffusion model, and maintains its general performance in Vincent image and Vincent video tasks; the present invention avoids the overfitting problem in fine-tuning through additional model design, and realizes dedicated optimization for specific editing tasks; the present invention takes into account the high efficiency and wide applicability of the original diffusion model, and provides highly customized editing functions in practical applications.

[0056] In addition, the application scenarios of the present invention may include: 1. Film and television post-production: The editing method of the present invention can achieve efficient lens modification, scene reconstruction and special effects generation in the film and television industry, and is particularly suitable for projects that need to complete a large number of editing tasks in a short period of time. In traditional film and television post-production, the modification of lenses and scenes usually requires a lot of manual intervention, which consumes a lot of time and energy. The present invention can significantly improve work efficiency, reduce manual operations, and quickly complete complex editing work through advanced automated editing technology. The application of this technology not only shortens the production cycle of the project, but also ensures that labor costs are reduced and productivity is improved while maintaining creative quality.

[0057] 2. Advertising and marketing content generation: Advertising companies can use the editing method of the present invention to quickly generate images and video content that meet specific advertising needs. By analyzing the user's editing preferences, market trends, and the needs of the target audience, personalized advertising materials can be customized. This process greatly shortens the advertising creation cycle, allowing advertising production to respond to market changes and consumer needs more quickly and flexibly. In addition, the present invention can help advertising companies quickly generate advertising materials of various specifications and styles on different media platforms, improve the efficiency and effectiveness of advertising delivery, and enhance the market adaptability of advertising creativity.

[0058] 3. Game development and design: During the game development process, especially when images or video materials need to be updated frequently, developers can quickly generate and adjust the images, scenes, and character materials in the game through the editing method of the present invention. This technology can ensure that the landscape, characters, and scene styles in the game are consistent with the overall game design and player needs, thereby improving the immersion of the game and the player experience. At the same time, the use of the automated editing method of the present invention can greatly reduce the workload of manual modeling and rendering, reduce development costs, and significantly shorten the game development cycle. The present invention is particularly suitable for a game development model with rapid iteration and testing, which improves development efficiency and flexibility.

[0059] 4. Personalized entertainment applications: In the field of personalized entertainment, such as customized video production, virtual idol creation, etc., the editing method of the present invention can quickly generate content and customize it to meet the diverse needs of users for entertainment products. Users can customize video content according to their personal preferences, select the appearance and behavioral characteristics of virtual idols, etc., to obtain a more personalized entertainment experience. Using this technology, entertainment products can not only be launched more quickly, but also achieve higher flexibility and diversity in creativity, meeting the growing demand of modern users for personalized content. At the same time, this technology can also promote innovation in the entertainment industry and open up new business models and application scenarios.

[0060] Furthermore, if Figure 3 As shown, based on the above-mentioned image and video reconstruction method based on the diffusion model, the present invention also provides an image and video reconstruction system based on the diffusion model, wherein the image and video reconstruction system based on the diffusion model includes: A forward diffusion processing module 51 is used to obtain original image data and original text description, and input the original image data and the original text description into a diffusion model to obtain noise data and intermediate features; A time step prediction model processing module 52 is used to determine a time step prediction model, and perform reverse generation processing on the noise data, the intermediate features and the original text description through the time step prediction model and the diffusion model to obtain initial image data; A time step prediction model updating module 53 is used to calculate a reconstruction error between the original image data and the initial image data, and to update the time step prediction model according to the reconstruction error to obtain an updated time step prediction model; The reverse generation processing module 54 is used to obtain the updated text description, and perform reverse generation processing on the noise data, the intermediate features and the updated text description through the updated time step prediction model and the diffusion model to obtain the target image data.

[0061] Furthermore, if Figure 4As shown, based on the above diffusion model-based image and video reconstruction method and system, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 4 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0062] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, the memory 20 stores an image and video reconstruction program 40 based on a diffusion model, and the image and video reconstruction program 40 based on a diffusion model can be executed by the processor 10, thereby realizing the image and video reconstruction method based on a diffusion model in the present application.

[0063] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the diffusion model-based image and video reconstruction method.

[0064] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0065] In one embodiment, when the processor 10 executes the diffusion model-based image and video reconstruction program 40 in the memory 20 , the steps of the diffusion model-based image and video reconstruction method are implemented.

[0066] In summary, the present invention provides an image and video reconstruction method, system and terminal based on a diffusion model, the method comprising: obtaining original image data and original text description, and inputting the original image data and the original text description into a diffusion model to obtain noise data and intermediate features; determining a time step prediction model, and performing reverse generation processing on the noise data, the intermediate features and the original text description through the time step prediction model and the diffusion model to obtain initial image data; calculating a reconstruction error between the original image data and the initial image data, and updating the time step prediction model according to the reconstruction error to obtain an updated time step prediction model; obtaining an updated text description, and performing reverse generation processing on the noise data, the intermediate features and the updated text description through the updated time step prediction model and the diffusion model to obtain target image data. The present invention uses a time step prediction model to assist the diffusion model in reverse generation, which can effectively improve the reconstruction capability of the diffusion model and reduce the reconstruction error of the diffusion model in the reverse generation process, thereby ensuring that the generated target image data is more consistent with the original image data, and effectively improving the reconstruction effect of the original image data.

[0067] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0068] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0069] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for image and video reconstruction based on a diffusion model, characterized in that: The image and video reconstruction method based on the diffusion model includes: Acquire original image data and original text description, and input the original image data and the original text description into a diffusion model to obtain noise data and intermediate features; Determine a time step prediction model, and perform reverse generation processing on the noise data, the intermediate features, and the original text description through the time step prediction model and the diffusion model to obtain initial image data; Calculating a reconstruction error between the original image data and the initial image data, and updating the time step prediction model according to the reconstruction error to obtain an updated time step prediction model; An updated text description is obtained, and the noise data, the intermediate features and the updated text description are subjected to reverse generation processing through the updated time step prediction model and the diffusion model to obtain target image data.

2. The image and video reconstruction method based on diffusion model according to claim 1, characterized in that: The obtaining of original image data and original text description, and inputting the original image data and the original text description into a diffusion model to obtain noise data and intermediate features specifically includes: Acquire original image data and original text description, and input the original image data and the original text description into a diffusion model; Performing forward diffusion processing on the original image data and the original text description through the diffusion model to obtain noise data; A preset time step is determined, and a key-value cache of an attention mechanism in the diffusion model is extracted according to the preset time step, and an intermediate feature corresponding to the preset time step is obtained according to the key-value cache.

3. The image and video reconstruction method based on diffusion model according to claim 1, characterized in that: The determining of the time step prediction model and performing reverse generation processing on the noise data, the intermediate features and the original text description through the time step prediction model and the diffusion model to obtain the initial image data specifically includes: Inputting the noise data, the intermediate features and the original text description into the diffusion model, and performing reverse generation processing on the noise data, the intermediate features and the original text description through the diffusion model; Determine a time step prediction model, and perform prediction processing on each time step in the reverse generation process through the time step prediction model; When any time step in the reverse generation process is 0, the reverse generation process is completed, and the diffusion model outputs initial image data.

4. The image and video reconstruction method based on diffusion model according to claim 1, characterized in that: The calculating a reconstruction error between the original image data and the initial image data, and updating the time step prediction model according to the reconstruction error to obtain an updated time step prediction model, specifically includes: Calculating a reconstruction error between the original image data and the initial image data, and determining whether the reconstruction error is greater than or equal to a preset threshold; If so, the time step prediction model is updated to obtain an updated time step prediction model.

5. The image and video reconstruction method based on diffusion model according to claim 4, characterized in that: The updating of the time step prediction model to obtain an updated time step prediction model specifically includes: Obtaining a negative number of the reconstruction error, and using the negative number as a reward signal; Historical generation data is obtained, and the time step prediction model is updated according to the historical generation data and the reward signal to obtain an updated time step prediction model.

6. The image and video reconstruction method based on diffusion model according to claim 5, characterized in that: The step of calculating a reconstruction error between the original image data and the initial image data, and updating the time step prediction model according to the reconstruction error to obtain an updated time step prediction model, further comprises: The noise data, the intermediate features and the original text description are iteratively reversely generated by using the update time step prediction model and the diffusion model until the reconstruction error is less than the preset threshold.

7. The image and video reconstruction method based on diffusion model according to claim 1, characterized in that: The acquiring of the updated text description and performing reverse generation processing on the noise data, the intermediate features and the updated text description through the updated time step prediction model and the diffusion model to obtain the target image data specifically includes: Determine user needs, and perform text update processing on the original text description according to the user needs to obtain an updated text description, wherein the text update processing includes keyword replacement and content detail modification; The updated text description, the noise data and the intermediate features are input into the diffusion model again, and the updated text description, the noise data and the intermediate features are reversely generated through the updated time step prediction model and the diffusion model to obtain target image data.

8. An image and video reconstruction system based on a diffusion model, characterized in that: The image and video reconstruction system based on the diffusion model includes: A forward diffusion processing module, used for acquiring original image data and original text description, and inputting the original image data and the original text description into a diffusion model to obtain noise data and intermediate features; A time step prediction model processing module is used to determine a time step prediction model, and perform reverse generation processing on the noise data, the intermediate features and the original text description through the time step prediction model and the diffusion model to obtain initial image data; A time step prediction model updating module, used for calculating a reconstruction error between the original image data and the initial image data, and updating the time step prediction model according to the reconstruction error to obtain an updated time step prediction model; The reverse generation processing module is used to obtain the updated text description, and perform reverse generation processing on the noise data, the intermediate features and the updated text description through the updated time step prediction model and the diffusion model to obtain the target image data.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a diffusion model-based image and video reconstruction program stored in the memory and executable on the processor. When the diffusion model-based image and video reconstruction program is executed by the processor, the steps of the diffusion model-based image and video reconstruction method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a diffusion model-based image and video reconstruction program, which, when executed by a processor, implements the steps of the diffusion model-based image and video reconstruction method according to any one of claims 1 to 7.

Images